As you are no doubt aware, Google has now started to roll-out its generative AI powered search results to more users.
In my opinion, the introduction of SGE will be the biggest shake up in the SEO world in the past two decades.
Here's some of our latest research to prove my point:
SGE Research on eCommerce terms.
SGE Research on Brand terms.
Over the past few months we have been developing our APIs and platform to ensure we are ready to give our clients the most up-to-date view of the SERPs the moment SGE rolls out globally for all users.
If you just want to continue rank tracking as usual, then there are no breaking changes to the structure of the JSON and you can ignore the rest of this help page.
If you want to understand what the SERP looks like after Google injects the generative AI snippet and URLs into the top of the SERP then read on.
N/B:
If you are an existing paying SERPs API subscriber then you can prepare your systems now to request SGE results, but you cannot use this method in production until Google has released SGE to non-logged in users.
We will notify you as soon as we make this available. Please bear in mind, Google may choose to do a phased rollout and SGE may only be available in certain languages and markets.
New Parameter to request SGE Jobs
There is a new option to request that we expand the SGE element and extract the generative snippet and ranking pages.
To request an SGE-enabled SERPs Job, simply add 'include_sge' in your SERPs API request JSON.
"include_sge": true
If this is requested, we make a SERPs request as usual and get the initial page of results ('universal_initial').
We then detect whether an SGE trigger (call-to-action) is present or whether the SGE is auto-generated (see image below).
If there is no SGE button or link to click, or, the SGE results are not auto-generated, then the SERP results will be paginated and parsed as usual and the new JSON objects will be empty.
If we detect an SGE trigger, we will click on it and wait until Google has injected the new generative AI snippet and URLs.
We will then extract and parse these new results into the JSON objects set out below and paginate accordingly until we reach the maximum number of results you have specified.
New JSON Objects for SGE in the SERPs API Response
If we find generative results in the SERPs then the following objects will be populated, otherwise they will be empty.
'generative_trigger' - This shows the type of SGE response we found. This is useful to know, as there are three different types (so far) and they take up differing amounts of space on the SERP and so will have different click rates.
'sub_type'will be set to 'auto', 'generate_button' or 'show_more'.
"generative_trigger": {"1": {"page_number": 1, "visible": true, "markup": 1, "sub_type": "generate_button", "above_the_fold": true, "description": null, "top_left": "230,301", "bottom_right": "1905,2013", "type": "generative_trigger", "title": null, "url": null, "content": "Chicago, Illinois has many places to get pizza, including casual spots, places with Chicago-style pizza, and other pizza places."}}
'generative' - This shows the pages ranking in the generative results. We try and identify different types of SGE results and sub_types and parse them according to how they are found on the SERPs page.
The default text links and citations are given a 'sub_type' of 'NULL'.
"generative": {"1": {"page_number": 1, "visible": true, "markup": 1, "sub_type": null, "above_the_fold": true, "description": null, "top_left": "1204,369", "bottom_right": "1330,587", "position": "right", "type": "generative", "title": "THE 10 BEST PIZZA DELIVERY in Chicago 2024 | Order Pizza Near ...", "url": "https://www.ubereats.com/category/chicago-il/pizza#:~:text=Pizza%20Delivery%20in%20Chicago%20%C2%B7%20Domino's%20Pizza,Grill%20(Clark%20St)%20%C2%B7%20Sbarro%20(333%20S."}}
Product listings have a 'sub_type' of 'product' - see examples below.
Local SGE results have a 'sub_type' of 'place' - see examples below.
There are a variety of product lists, grids and carousels that appear within the main SGE snippet. These lists vary in size from 3 rows and 1 column, to a 1 x 3 (carousel pack), 3 x 3 grid, 1 x 10 grid and 5 x 5 grids.
As they are part of the main SGE snippet and are often topped and tailed with text and citations, you will find them in the 'generative' object with a 'sub_type' of 'product' (see examples below).
This is likely to change as Google iterates its designs for integrating generative AI into Search.
Generative URLs are ordered using the x,y coordinates ('top_left', 'bottom_right') with SGE elements closer to the top and left ranked highest.
'place' - this lists up to 5 (this is the most we've seen so far) local business listings, places or locations. Note, Google is not showing the URL of the business, but you can build a lookup table for known locations using Google's unique identifier for each local entity - see 'cid'.
We have also added a few extra data points that will be helpful for analysis and optimisation:
N/B: 'sub_type' will be set to 'null' (for standard text listings and unidentified sub-types) or 'product' or 'place'.
'position' will be set to 'left', 'right'.
Whilst not new, these key value pairs will be useful for further SEO analysis:
'visible' will be set to 'true', 'false'.
'top_left' shows the x,y coordinates for the item and it will be useful to understand how big the SGE results are and how far down the page your organic results have been pushed.
'generative_follow_up_question' - contains a list of up to 3 frequently asked questions.
"generative_follow_up_question": {"1": {"page_number": 1, "visible": true, "markup": 1, "above_the_fold": false, "description": null, "top_left": "230,1868", "bottom_right": "1330,1914", "type": "generative_follow_up_question", "title": null, "items": {"1": {"title": "Which pizza place offers the best deals?"}, "2": {"title": "Which pizza place has the best Chicago-style pizza?"}, "3": {"title": "What are some affordable pizza places in Chicago?"}}, "url": null}}
'univeral_initial' - this reflects the state of the SERP prior to the expansion of the SGE element and only appears for SGE-enabled jobs.
'universal' - this reflects the state of the SERP for standard non SGE-enabled jobs and after the expansion of the SGE element for SGE-enabled jobs. The JSON is structured the same way as the 'universal_initial' pictured above.
All other objects - reflect the state of the SERP for standard non SGE-enabled jobs and after the expansion of the SGE element for SGE-enabled jobs.
SERPs API Billing & Reporting Changes
The new SGE results require more effort and resources to get, as such they will be charged at 2x the cost of a standard query. This will be reflected in your usage and billing emails and invoices.
We will review the costs associated with providing SGE results in response to demand and supply considerations (like reliability and consistency of results) and in light of how Google eventually rolls this out to all users in different markets and languages.
It's possible that Google may eventually roll out SGE automatically in the SERPs for all queries, so in time the need for this additional parameter and the additional work and costs may eventually be unnecessary.
SGE - Initially only available on Standard Priority
Initially, SGE results will only be available on the 'Standard' priority jobs. We do plan to add support for higher priority jobs post-launch and will advise you of this in due course.
Tips:
We have noticed that search engines treats requests for 100 results with greater suspicion these days, as it seems only SEOs paginate or scroll this far. We expect this will continue to be the case for the new generative SERPs. We therefore recommend setting the maximum number of results to a number between 30 and 50 for all SERPs API requests. This is likely to make your requests look less like a bot and more like a human - and decreases the chance of your requests bots getting blocked by the search engines.
When SGE is clicked the organic results are just pushed down the page. So far we have not observed any changes in the make-up or order of the ranking organic URLs. So, rather than make two separate requests for the same keyword with SGE enabled and disabled. You could make just one request with SGE enabled and then compare the
'universal_intial'with type'organic'against the'universal'results with type'organic'to see how far the top ranking organic results have fallen using differences in the y position extracted from the'top_left'x, y coordinates. Please note, the'universal_intial'only contains the first page Google shows us, so it will not have a complete list of all organic results. To see the complete list use'universal'with type set to'organic'.
FAQs:
Q. What happens if SGE is auto-generated for a keyword and the client has not set the parameter for us to click on the SGE trigger?
A. There will be no 'universal_initial', we will not expand the SGE elements and will only capture generative content that's visible without user interaction on the initial page or in the HTML.
Footnote:
Please note that SGE is being rolled out in phases and the types of results and content displayed is changing quite often. We will update this page regularly to reflect changes we observe.