parse-robots

Parses the robots.txt file to an abstract syntax tree.

Parses robots.txt files to abstract syntax trees.

ParseRobots.prototype.sources(file, context)

Parse a robots.txt file to an abstract syntax tree, the parsed AST is assigned to file.ast.robots.

Represents a parsed line in a robots.txt file.

new RobotsLine(line, index[, key][, value])

Create a RobotsLine.

String key

The declaration key.

String value

The declaration value.

readonly String comment

A comment on this line.

readonly String line

The raw input line.

readonly Number lineno

The line number using a one-based index.

RobotsLine.prototype.hasComment()

Determine if this line has a commment.

Returns a boolean indicating if this line contains a comment.

RobotsLine.prototype.hasKeyPair()

Determine if this line has a valid key/value pair.

Returns a boolean indicating if this line contains a key and value.

RobotsLine.prototype.parse()

Parse the line into this instance.

Returns this line instance.

RobotsLine.prototype.serialize()

Get a serialized line from the current state.

Returns a string line value.

Parse and serialize a robots.txt file.

Designed so that the serialized output has a 1:1 relationship with the source document but allows inspecting and modifying the key and value properties for each line.

RobotsParser.prototype.parse(content)

Parse the robots.txt file content.

User-Agent:
Disallow: /private/ # does not block indexing, add meta noindex

Becomes:

[
  {
    key: 'User-Agent',
    value: '*',
    lineno: 1,
    line: 'User-Agent: *'
  },
  {
    key: 'Disallow',
    value: '/private/',
    lineno: 2,
    line: 'Disallow: /private/ # does not block indexing, add meta noindex',
    comment: '# does not block indexing, add meta noindex'
  }
]

Returns an array of line objects.

RobotsParser.prototype.serialize(list)

Serialize the robots.txt declaration list.

Returns a string of robots.txt file content.