For each type of file, you can specify whether the contents of the file are indexed. true or false?

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.

Table of Contents Show

What is a robots.txt file used for?
Understand the limitations of a robots.txt file
Advanced options
Q: How does indexing work?
Q: Can I use source indexing on a portable PDB created from a .NET Core assembly?
Q: How long are Symbols retained?
Related articles

If you use a CMS, such as Wix or Blogger, you might not need to (or be able to) edit your robots.txt file directly. Instead, your CMS might expose a search settings page or some other mechanism to tell search engines whether or not to crawl your page.

If you want to hide or unhide one of your pages from search engines, search for instructions about modifying your page visibility in search engines on your CMS (for example, search for "wix hide page from search engines").

What is a robots.txt file used for?

A robots.txt file is used primarily to manage crawler traffic to your site, and usually to keep a file off Google, depending on the file type:

robots.txt effect on different file types

Web page

You can use a robots.txt file for web pages (HTML, PDF, or other non-media formats that Google can read), to manage crawling traffic if you think your server will be overwhelmed by requests from Google's crawler, or to avoid crawling unimportant or similar pages on your site.

Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results.

If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex.

If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result will not have a description. Image files, video files, PDFs, and other non-HTML files will be excluded. If you see this search result for your page and want to fix it, remove the robots.txt entry blocking the page. If you want to hide the page completely from Search, use another method.

Media file

Use a robots.txt file to manage crawl traffic, and also to prevent image, video, and audio files from appearing in Google search results. This won't prevent other pages or users from linking to your image, video, or audio file.

Resource file

You can use a robots.txt file to block resource files such as unimportant image, script, or style files, if you think that pages loaded without these resources will not be significantly affected by the loss. However, if the absence of these resources make the page harder for Google's crawler to understand the page, don't block them, or else Google won't do a good job of analyzing pages that depend on those resources.

Understand the limitations of a robots.txt file

Before you create or edit a robots.txt file, you should know the limits of this URL blocking method. Depending on your goals and situation, you might want to consider other mechanisms to ensure your URLs are not findable on the web.

robots.txt directives may not be supported by all search engines.
The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them. While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not. Therefore, if you want to keep information secure from web crawlers, it's better to use other blocking methods, such as password-protecting private files on your server.
Different crawlers interpret syntax differently.
Although respectable web crawlers follow the directives in a robots.txt file, each crawler might interpret the directives differently. You should know the proper syntax for addressing different web crawlers as some might not understand certain instructions.
A page that's disallowed in robots.txt can still be indexed if linked to from other sites.
While Google won't crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google search results. To properly prevent your URL from appearing in Google search results, password-protect the files on your server, use the noindex meta tag or response header, or remove the page entirely.

Caution: Combining multiple crawling and indexing directives might cause some directives to counteract other directives. Learn how to combine crawling with indexing and serving directives.

If you decided that you need one, learn how to create a robots.txt file. Or if you already have one, learn how to update it.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2022-07-04 UTC.

[{ "type": "thumb-down", "id": "missingTheInformationINeed", "label":"Missing the information I need" },{ "type": "thumb-down", "id": "tooComplicatedTooManySteps", "label":"Too complicated / too many steps" },{ "type": "thumb-down", "id": "outOfDate", "label":"Out of date" },{ "type": "thumb-down", "id": "samplesCodeIssue", "label":"Samples / code issue" },{ "type": "thumb-down", "id": "otherDown", "label":"Other" }] [{ "type": "thumb-up", "id": "easyToUnderstand", "label":"Easy to understand" },{ "type": "thumb-up", "id": "solvedMyProblem", "label":"Solved my problem" },{ "type": "thumb-up", "id": "otherUp", "label":"Other" }]

Article
07/26/2022
5 minutes to read

Azure DevOps Services | Azure DevOps Server 2020 | Azure DevOps Server 2019 | TFS 2018

Use this task to index your source code and publish your symbols to a file share or Azure Artifacts symbol server.

Indexing your source code allows you to use your symbol files to debug your application on a machine other than the one you used to build your application. For example, you can debug an application built by a build agent from a dev machine that does not have the source code.

Symbol servers enables your debugger to automatically retrieve the correct symbol files without knowing product names, build numbers, or package names.

Demands

None

# Index sources and publish symbols # Index your source code and publish symbols to a file share or Azure Artifacts symbol server - task: PublishSymbols@2 inputs: #symbolsFolder: '$(Build.SourcesDirectory)' # Optional #searchPattern: '**/bin/**/*.pdb' #indexSources: true # Optional #publishSymbols: true # Optional #symbolServerType: ' ' # Required when publishSymbols == True # Options: , teamServices, fileShare #symbolsPath: The file share that hosts your symbols. # Optional #compressSymbols: false # Required when symbolServerType == FileShare #SymbolExpirationInDays: The number of days that symbols should be retained. # Required when PublishSymbols = true && SymbolServerType = TeamServices. Default: 36530 #IndexableFileFormats: Which debug formats to publish to the symbol server. # Options: Default, Pdb, SourceMap, All. # Required when PublishSymbols = true && SymbolServerType = TeamServices. Default: Default #detailedLog: true # Optional #treatNotIndexedAsWarning: false # Optional #useNetCoreClientTool: false # Optional #symbolsMaximumWaitTime: # Optional #symbolsProduct: # Optional #symbolsVersion: # Optional #symbolsArtifactName: 'Symbols_$(BuildConfiguration)' # Optional

Arguments

Argument	Description
SymbolsFolder Path to symbols folder	(Optional) The path to the folder that is searched for symbol files. The default is $(Build.SourcesDirectory), Otherwise specify a rooted path. Note that UNC paths aren't supported if you select the Azure Artifacts symbol server as the server type. For example: $(Build.BinariesDirectory)/MyProject
SearchPattern Search pattern	(Required) File matching pattern(s) The pattern used to discover the pdb files to publish Default value: /bin//*.pdb
IndexSources Index sources	(Optional) Indicates whether to inject source server information into PDB files. This option is only supported on Windows agents. Default value: true
PublishSymbols Publish symbols	(Optional) Indicates whether to publish the symbol files Default value: true
SymbolServerType Symbol server type	(Required) Choose where to publish symbols. Symbols published to the Azure Artifacts symbol server are accessible by any user with access to the organization/collection. Azure DevOps Server only supports the "File share" option. Follow these instructions to use Symbol Server in Azure Artifacts. TeamServices: Symbol Server in this organization/collection (requires Azure Artifacts) File share: Select this option to use the file share supplied in the next input.
SymbolsPath Path to publish symbols	(Optional) The file share that hosts your symbols. This value will be used in the call to symstore.exe add as the /s parameter. To prepare your SymStore symbol store: Set up a folder on a file-sharing server to store the symbols. For example, set up \fabrikam-share\symbols. Grant full control permission to the build agent service account. If you leave this argument blank, your symbols will be source indexed but not published. (You can also store your symbols with your drops. See Publish Build Artifacts).
CompressSymbols Compress symbols	(Required) Only available when File share is selected as the Symbol server type. Compresses your pdbs to save space. Default value: false
SymbolExpirationInDays Symbol Expiration (in days)	The number of days that symbols should be retained. Required when PublishSymbols = true && SymbolServerType = TeamServices. Default value: 36530

Advanced options

Argument	Description
IndexableFileFormats Symbol file formats to publish	Which debug formats to publish to the symbol server. Required when PublishSymbols = true && SymbolServerType = TeamServices. Options: Default, Pdb, SourceMap, All. Default value: Default
DetailedLog Verbose logging	(Optional) Enables additional log details. Default value: true
TreatNotIndexedAsWarning Warn if not indexed	(Optional) Indicates whether to warn if sources are not indexed for a PDB file. Otherwise the messages are logged as normal output. A common cause of sources to not be indexed are when your solution depends on binaries that it doesn't build. Even if you don't select this option, the messages are written in log. Default value: false
UseNetCoreClientTool Use version of tool that supports Linux symbols	(Optional) Uses a version of the symbol upload tool that supports DWARF and ELF files. This option only matters on Windows agents. On non-Windows agents, the version of the symbol upload tool that supports DWARF and ELF files will always be used. Default value: false
SymbolsMaximumWaitTime Max wait time (min)	(Optional) The number of minutes to wait before failing this task. If you leave it blank, limit is 2 hours.
SymbolsProduct Product	(Optional) Specify the product parameter to symstore.exe. The default is $(Build.DefinitionName)
SymbolsVersion Version	(Optional) Specify the version parameter to symstore.exe. The default is $(Build.BuildNumber).
SymbolsArtifactName Artifact name	(Optional) Specify the artifact name to use for the Symbols artifact. The default is Symbols_$(BuildConfiguration). Default value: Symbols_$(BuildConfiguration)

Important

To delete symbols that were published using the Index Sources & Publish Symbols task, you must first delete the build that generated those symbols. This can be accomplished by using retention policies or by manually deleting the run.

FAQ

Q: How does indexing work?

A: By choosing to index the sources, an extra section will be injected into the PDB files. PDB files normally contain references to the local source file paths only E.g: C:\BuildAgent_work\1\src\MyApp\Program.cs. The extra section injected into the PDB file contains mapping instructions for debuggers. The mapping information indicates how to retrieve the server item corresponding to each local path.

The Visual Studio debugger will use the mapping information to retrieve the source file from the server. An actual command to retrieve the source file is included in the mapping information. Example:

tf.exe git view /collection:http://SERVER:8080/tfs/DefaultCollection /teamproject:"93fc2e4d-0f0f-4e40-9825-01326191395d" /repository:"647ed0e6-43d2-4e3d-b8bf-2885476e9c44" /commitId:3a9910862e22f442cd56ff280b43dd544d1ee8c9 /path:"/MyApp/Program.cs" /output:"C:\Users\username\AppData\Local\SOURCE~1\TFS_COMMIT\3a991086\MyApp\Program.cs" /applyfilters

Q: Can I use source indexing on a portable PDB created from a .NET Core assembly?

A: No, but you can use Source Link instead.

Q: How long are Symbols retained?

A: Symbols are associated with the build that published to Azure Pipelines they are associated with a build. When the build is deleted either manually or using retention policies, the symbols are also deleted. If you want to retain the symbols indefinitely, mark the build as Retain Indefinitely.