“Towards a Distributed Search Engine”

22. November 2010

“Towards a Distributed Search Engine”

Created by Schahram Dustdar

The Distributed Systems Group of the Information Systems Institute invites to the following talk: “Towards a Distributed Search Engine”

Ricardo Baeza-Yates (Foto: http://www.dcc.uchile.cl/~rbaeza)
Ricardo Baeza-Yates

Lecturer: Ricardo Baeza-Yates
VP of Yahoo! Research for Europe and Latin America, leading the labs at Barcelona, Spain and Santiago, Chile
<link http: www.dcc.uchile.cl _blank link_intern>

www.dcc.uchile.cl/~rbaeza, opens an external URL in a new window

Date: Tuesday, 30 November 2010
Time: 11:00 h
Place: EI 10, Gußhausstraße 27-29, ground floor

Abstract:
In the ocean of Web data, Web search engines are the primary way to access content. As the data is on the order of petabytes, current search engines are very large centralized systems based on replicated clusters. Web data, however, is always evolving. The number of Web sites continues to grow rapidly (Over 200 millions nowadays) and there are currently more than 20 billion indexed pages. On the other hand, Internet users are above one billion and hundreds of million of queries are issued each day. In the near future, centralized systems are likely to become less effective against such a data-query load, thus suggesting the need of fully distributed search engines. Such engines need to maintain high quality answers, fast response time, high query throughput, high availability and scalability; in spite of network latency and scattered data. In this talk we present the main challenges behind the design of a distributed Web retrieval system and our research in all the components of a search engine: crawling, indexing, and query processing.

Bio:
Ricardo Baeza-Yates is VP of Yahoo! Research for Europe and Latin America, leading the labs at Barcelona, Spain and Santiago, Chile. Previously full professor at Univ. of Chile and ICREA research professor at UPF in Barcelona. Co-author of Modern Information Retrieval (Addison-Wesley, 1999) among other books and publications. Member of the ACM (Fellow), AMS, IEEE (Senior), SIAM and SCCC, as well as the Chilean Academy of Sciences. Awards from American Organization States, Institute of Engineers of Chile, and COMPAQ. His research interests includes algorithms and data structures, information retrieval, web mining, text and multimedia databases, software and database visualization, and user interfaces.

Organized by:
Schahram Dustdar
Full Professor
Vienna University of Technology, Austria
Distributed Systems Group (DSG), Information Systems Institute
A-1040 Vienna, Argentinierstrasse 8/184-1
T: +43-1-58801-18414
F: +43-1-58801-18491
<link http: www.infosys.tuwien.ac.at staff sd>

www.infosys.tuwien.ac.at/Staff/sd/, opens an external URL in a new window

Name	Purpose	Lifetime	Type	Provider
CookieConsent	Saves your settings for the use of cookies on this website.	1 year	HTML	Homepage TU Wien
SimpleSAML	This is needed to distinguish between the sessions of the logged-in users.	session	HTTP	Login TU Wien
SimpleSAMLAuthToken	This is needed to distinguish between the sessions of the logged-in users.	session	HTTP	Login TU Wien
fe_typo_user	Is needed so that in case of a Typo3 frontend login the session ID is recognized to grant access to protected areas.	session	HTTP	Homepage TU Wien
staticfilecache	Is needed to optimize the delivery time of the website.	session	HTTP	Homepage TU Wien
JESSIONSID	Is needed so that in case of a LectureTube the session ID is recognized to grant access to protected areas.	session	HTTP	LectureTube TU Wien
_shibsession_lecturetube	This is needed to distinguish between the sessions of the logged-in users.	session	HTTP	LectureTube TU Wien

Name	Purpose	Lifetime	Type	Provider
_pk_id	Used to store a few details about the user such as the unique visitor ID.	13 months	HTML	Matomo TU Wien
_pk_ref	Is used to store the information of the users home website.	6 months	HTML	Matomo TU Wien
_pk_ses	Is needed to store temporary data of the visit.	30 minutes	HTML	Matomo TU Wien
nmstat	Is used to record the behaviour on the website. It is used to collect statistics about website usage, such as when the visitor last visited the website. The cookie does not contain any personal data and is only used for website analysis.	1000 days	HTML	Siteimprove
siteimproveses	Is used to track the sequence of pages that a visitor views during his/her visit to the website. The cookie does not contain any personal data and is used solely for website analysis.	session	HTTP	Siteimprove
AWSELB	Always occurs in pairs with siteimproveses (for load balancing on the provider server)	session	HTTP	Siteimprove

Name	Purpose	Lifetime	Type	Provider
_ga	Is needed to distinguish the sessions of the users from each other.	persistent	HTTP	Google Analytics
_gali	Is needed to determine which links are clicked on a page.	expires immediately	HTTP	Google Analytics
_gat	This is a function-related cookie, whose tasks may differ.	2 years	HTTP	Google Analytics
_gid	Is needed to distinguish users and create statistics.	24 hours	HTTP	Google Analytics
_gads	Required to enable websites to display advertising from Google, including personalized advertising.	13 months	HTTP	Google Analytics
_gac_	Required by advertisers to measure user activity and the performance of their advertising campaigns.	90 days	HTTP	Google Analytics
_gcl_	Required by advertisers to determine how often users who click on their ads end up taking an action on their website.	90 days	HTTP	Google Analytics
_gcl_au	Contains a randomly generated user ID.	90 days	HTTP	Google
_gcl_aw	Is set when users click on a Google ad on the website and contains information about which ad was clicked.	90 days	HTTP	Google
__utma	Is used to record visits and visitors.	2 years	HTTP	Google Analytics
__utmb	Is used to detect new visits.	30 minutes	HTTP	Google Analytics
__utmc	Is used in connection with __utmb to determine whether it is a new (recent) visit.	session	HTTP	Google Analytics
__utmd	Is used to store and track visitor journeys through the site and classifies them into groups (marketing/tracking).	1 second	HTTP	Google Analytics
__utmt	Is needed to limit the query rate on Google Analytics.	10 minutes	HTTP	Google Analytics
__utmz	Is needed to determine from which source/campaign visitors come.	6 months	HTTP	Google Analytics
__utmvc	Is needed to collect information about user behavior on multiple websites. This information is used to optimize the relevance of advertising on the website.	24 hours	HTTP	Google AdSense
utm_source	Is needed to tag URLs with parameters to identify the campaigns that forward traffic.	expires immediately	HTTP	Google Analytics
__utm.gif	Is needed to save browser details.	session	HTTP	Google Analytics
gtag	Is needed to perform remarketing.	30 days	HTTP	Google AdSense
id	Is needed to perform remarketing.	2 years	HTTP	Google AdWords
1P_JAR	Is needed to optimize advertising, provide ads that are relevant to users, improve campaign performance reports, or prevent users from seeing the same ads more than once.	2 years	HTTP	Google
AID	Is needed to activate targeted advertising.	2 years	HTTP	Google Analytics
ANID	Is needed to display Google ads on non-Google websites.	2 years	HTTP	Google AdSense
APISID	Unknown functionality	2 years	HTTP	Google Ads Optimization
AR	Is needed to profile visitors' interests and display relevant ads on other websites. This cookie works by uniquely identifying your browser and device.	2 years	HTTP	Google AdSense
CONSENT	Is needed to store the preferences of visitors and personalize advertising.	persistent	HTTP	Google
DSID	Is needed by DoubleClick for advertising displayed in various places on the web and used to store the preferences of users.	2 years	HTTP	Doubleclick
DV	Is needed to store user preferences and other information. This includes, in particular, the preferred language, the number of search results to be displayed on the page, and the decision whether or not to activate the Google SafeSearch filter.	2 years	HTTP	Google
HSID	Contains the Google account ID and the last login time of the user.	2 years	HTTP	Google
IDE	Is needed by DoubleClick to record and report the actions of users on the website after viewing or clicking on one of the provider's ads, with the purpose of measuring the effectiveness of an advertisement and displaying targeted advertisements to users.	2 years	HTTP	Doubleclick
LOGIN_INFO	Is used to store the credentials of users of Google services.	2 years	HTTP	Google
NID	Is used to store information about user settings.	6 months	HTTP	Google
OTZ	Is needed to link activities of visitors with other devices that are previously logged in via the Google account. In this way, advertising is tailored to different devices.	1 month	HTTP	Google
RUL	Is needed by DoubleClick to determine whether advertising has been displayed correctly in order to make marketing activities more efficient.	1 year	HTTP	Doubleclick
SAPISID	Is needed by YouTube to store user settings and to calculate user bandwidth.	persistent	HTTP	Google
SEARCH_SAMESITE	Enables servers to mitigate the risk of CSRF and information leakage attacks by specifying that a particular cookie may only be sent on requests originating from the same registerable domain.	6 months	HTTP	Google
SID	Contains the Google account ID and the last login time of the user.	2 years	HTTP	Google
SIDCC	Is needed to store information about user settings and information for Google Maps.	3 months	HTTP	Google
SSID	Is needed to collect visitor information for videos hosted by YouTube on Google Maps integrated maps.	persistent	HTTP	Google
__SECURE-1PAPISID	Is needed for targeting purposes to create a profile of the interests of website visitors.	2 years	HTTP	Google
__SECURE-1PSID	Is needed for targeting purposes to create a profile of the interests of website visitors.	2 years	HTTP	Google
__SECURE-3PAPISID	Is needed for targeting purposes to create a profile of the interests of website visitors.	2 years	HTTP	Google
__SECURE-3PSID	Is needed for targeting purposes to create a profile of the interests of website visitors.	2 years	HTTP	Google
__SECURE-3PSIDCC	Is needed for targeting purposes to create a profile of the interests of website visitors.	2 years	HTTP	Google
__SECURE-APISID	Is needed to profile the interests of website visitors in order to display relevant and personalized advertising through retargeting.	8 months	HTTP	Google
__SECURE-HSID	Is needed to secure digitally signed and encrypted data from the unique Google ID and to store the last login time that Google uses to identify visitors, prevent fraudulent use of login data, and protect visitor data from unauthorized parties. This may also be used for targeting purposes to display relevant and personalized advertising content.	8 months	HTTP	Google
__SECURE-SSID	Is needed to store information about how visitors use the site and about the ads they may have seen before visiting the site. Also used to customize ads on Google domains.	8 months	HTTP	Google
test_cookie	Is set as a test to check whether the browser allows cookies to be set. Does not contain any identification features.	15 minutes	HTTP	Google
VISITOR_INFO1_LIVE	Is needed by YouTube to store user settings and to calculate user bandwidth.	6 months	HTTP	Youtube
facebook	Is used to Enable ad delivery or retargeting	90 days	HTTP	Meta (Facebook)
__fb_chat_plugin	Is needed to store and track interactions (marketing/tracking).	persistent	HTTP	Meta (Facebook)
_js_datr	Is needed to save user settings.	2 years	HTTP	Meta (Facebook)
_fbc	Is needed to save the last visit (marketing/tracking).	2 years	HTTP	Meta (Facebook)
fbm	Is needed to store account data (marketing/tracking).	1 year	HTTP	Meta (Facebook)
xs	Is needed to store a unique session ID (marketing/tracking).	1 year	HTTP	Meta (Facebook)
wd	Is needed to log the screen resolution.	1 week	HTTP	Meta (Facebook)
fr	Is needed to serve ads and measure and improve their relevance.	3 months	HTTP	Meta (Facebook)
act	Is needed to store logged in users (marketing/tracking).	90 days	HTTP	Meta (Facebook)
_fbp	Is needed to store and track visits to various websites (marketing/tracking).	3 months	HTTP	Meta (Facebook)
datr	Is needed to identify the browser for security and website integrity purposes, including account recovery and identification of potentially compromised accounts.	2 years	HTTP	Meta (Facebook)
dpr	Is used for analysis purposes. Technical parameters are logged (e.g. aspect ratio and dimensions of the screen) so that Facebook apps can be displayed correctly.	1 week	HTTP	Meta (Facebook)
sb	Is needed to store browser details and security information of the Facebook account.	2 years	HTTP	Meta (Facebook)
dbln	Is needed to store browser details and security information of the Facebook account.	2 years	HTTP	Meta (Facebook)
spin	Is needed for promotional purposes and social campaign reporting.	session	HTTP	Meta (Facebook)
presence	Contains the "chat" status of logged in users.	1 month	HTTP	Meta (Facebook)
cppo	Is needed for statistical purposes.	90 days	HTTP	Meta (Facebook)
locale	Is needed to save the language settings.	session	HTTP	Meta (Facebook)
pl	Required for Facebook Pixel.	2 years	HTTP	Meta (Facebook)
lu	Required for Facebook Pixel.	2 years	HTTP	Meta (Facebook)
c_user	Required for Facebook Pixel.	3 months	HTTP	Meta (Facebook)
bcookie	Is needed to store browser data (marketing/tracking).	2 years	HTTP	LinkedIn
li_oatml	Is needed to identify LinkedIn members outside of LinkedIn for advertising and analytics purposes.	1 month	HTTP	LinkedIn
BizographicsOptOut	Is needed to save privacy settings.	10 years	HTTP	LinkedIn
li_sugr	Is needed to store browser data (marketing/tracking).	3 months	HTTP	LinkedIn
UserMatchHistory	Is needed to provide advertising or retargeting (marketing/tracking).	30 days	HTTP	LinkedIn
linkedin_oauth_	Is needed to provide cross-page functionality.	session	HTTP	LinkedIn
lidc	Is needed to store performed actions on the website (marketing/tracking).	1 day	HTTP	LinkedIn
bscookie	Is needed to store performed actions on the website (marketing/tracking).	2 years	HTTP	LinkedIn
X-LI-IDC	Is needed to provide cross-page functionality (marketing/tracking).	session	HTTP	LinkedIn
AnalyticsSyncHistory	Stores the time when the user was synchronized with the "lms_analytics" cookie.	30 days	HTTP	LinkedIn
lms_ads	Is needed to identify LinkedIn members outside of LinkedIn.	30 days	HTTP	LinkedIn
lms_analytics	Is needed to identify LinkedIn members for analytics purposes.	30 days	HTTP	LinkedIn
li_fat_id	Required for indirect member identification used for conversion tracking, retargeting and analytics.	30 days	HTTP	LinkedIn
U	Is needed to identify the browser.	3 months	HTTP	LinkedIn
_guid	Is needed to identify a LinkedIn member for advertising via Google Ads.	90 days	HTTP	LinkedIn

News articles

“Towards a Distributed Search Engine”