Proposal API help

From MikroTik Wiki
Revision as of 21:00, 13 July 2015 by Boen robot (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Introduction

Abstract

The API protocol, unlike the command line (and by extension - the Telnet and SSH protocols), does not offfer a way for RouterOS to report to the client the available commands, menus, attributes or attribute values. This proposal suggests a new API protocol specific command called "/help" that behaves similarly to a "?" in the command line, in that it will fill this gap, and it should hopefully be relatively easy to implement by MikroTik due to its similarity with the "?" help.

Terminology

  • API protocol - The syntax defined in the corresponding manual page.
  • API client - A general purpose implementation of the API protocol for use in a programming language. All "Examples" at the end of the above manual page are essentially API clients.
  • (API) application - A program or script written in a programming language that uses an API client as part of its functionality.
  • Users - People using API clients inside their applications or otherwise interact directly with the API protocol.
  • End users - People for which the "Users" write the API applications.

Use cases

"Why would such a feature even be needed in a protocol intended for machine consumption?" you may ask. There are several notable use cases:

  • Utility methods in API clients may sometimes need to behave differently based on the presence or absence of a command at a particular menu. Case in point: The "move" command, based on which API clients respectively may support or should avoid supporting "numbers" emulation at the given menu.
  • API clients will be able to (if the user wants to) pre-validate commands (or at least some of the more trivial aspects of command calls), lessening the load on the router at the expense of the API application.
  • Applications could be written that generate classes (or the equivalent in whatever language the API protocol is used from) representing each menu/command, allowing IDEs to seamlessly assist users when creating API applications.
  • Because most commands are shared between scripting and API, one could also make an application that uses this command to generate auto-complete suggestions for ".rsc" files.
  • It lets applications potentially detect upgrades to RouterOS, and in turn verify automatically that there are no breaking changes as a result. Case in point: v5's "target-address"/"interface" combo in "/queue simple" vs. v6's "target".

Syntax

No arguments

When the "/help" command is called with no arguments, only a single "!done" reply should be given, with a "ret" attribute that has a hash value in it. The hash should be computed based on the names and build times of all installed and enabled RouterOS packages.

The actual hash algorithm may be changed, without any prior notice, from one RouterOS build to the next. The only thing that applications should be able to safely assume about it is that it would be different whenever RouterOS is upgraded/downgraded, or if a RouterOS package is installed/uninstalled/upgraded/downgraded/enabled/disabled.

Icon-note.png

Note: MD5 or SHA1 are good choices here, since collisions are not easy enough to create when the whole data is a small series of names and times, and there's no security concern here.


Icon-note.png

Note: Version numbers are intentionally avoided, as to allow seamless uses of improperly labeled RC versions of RouterOS or its packages.


Icon-warn.png

Warning: Applications MUST NOT assume ANY particular hashing algorithm or alphabet of the hash. When storing the hash, applications should take care to escape the value for whatever context they are storing it in.


Example:

/help
 !done
=ret=ebddd18303a54111e2dea05a92ab46b4

The purpose of this hash is to allow applications to "cache" the other results from "/help", particularly in cases where an application is examining the whole of RouterOS. For example, an object generator may generate different sets of objects for different RouterOS hostnames, but not bother regenerating them when there's no need to.

Enabling/Disabling a package affects the given hash, because enabling/disabling a package will more often than not affect the presence of menus, commands, attributes and attribute values, which is the primary use case for making an overall examination of RouterOS to begin with.

The "menu" argument

When the "/help" command is called with ONLY the "menu" argument, the "/help" command should return all menus and commands available immediatly from the specified menu.

The value of the "menu" argument is an absolute path to the menu, using the API protocol syntax, with the leading slash, and without a trailing slash. Applications can safely assume there's always going to be a root menu, i.e. "/", but should not assume the existence of any submenu.

The "!done" reply may optionally contain the following attributes:

  • description - A long description of the menu, as seen above the list of commands and menus when using "?" after a menu.

For each menu or command, a "!re" reply should be generated, with the following attributes:

  • name - The name of the menu or command.
  • summary - A short description of the menu/command, as seen in the help from "?" after each listed menu.
  • type - Indicates whether this is a menu or a command. Possible values are only "command" and "menu".
  • env - Optional. A comma separated list of protocols/environments from which the command/menu is available. When not present, applications may assume that this menu/command is available from all protocols/environments supported by that RouterOS version. Each value may also be preceeded by "!" to exclude it from a previously included protocol/environment. Possible values:
    • api - The API protocol.
    • cli - The command line, both local and from protocols like Telnet and SSH. Also includes scripting.
    • script - Scripting. Both in ".rsc" files and in "/system script" scripts and the like. Does not include the command line, Telnet and SSH.
  • policy - A comma separated list of permissions that this menu/command requires. In the case of commands, this list includes the required permissions when only the unconditionally required arguments are present. In the absence of this attribute, it is assumed that the menu/command is always available.
Icon-note.png

Note: The "env" attribute is defined in this way, so as to allow forwards compatibility in case new protocols are added with their own restrictions or if commands are shifted in seemingly counter intuitive ways, e.g. not being available from scripting but being available from the command line or vice-versa. New values can be added on an "as needed" basis to exclude previously included values.


Icon-note.png

Note: Some commands' availablility over a protocol/environment is dependent on the presence/absence/value of certain arguments. RouterOS must consider the command as available to a protocol/environment if there's ANY set of arguments for which that command could work over that protocol. In other words, a command that only works under "cli" - except when a certain argument is provided and/or has a certain value - is NOT considered "cli" specific, because of that specific case. Case in point: The export command, which is really only useful when a "file" argument is provided.


Icon-warn.png

Warning: Command line specific navigation features, notably ".." and ":", are excluded from the list of available menus, due to it being impossible to reliably detect them as "not-exactly-menus-on-their-own" without special casing the application code. If an application wants to support them, it would end up doing that anyway, due to these features' behaviour being special. There's no need force other applications to also special case these, so they're excluded for their benefit.


Example:

/help
=menu=/
 !re
=name=beep
=summary=
=type=command
 !re
=name=cancel
=summary=Cancel a running command
=type=command
=env=api
 !re
=name=certificate
=summary=Certificate management
=type=menu
 !re
=name=console
=summary=
=type=menu
 !re
=name=delay
=summary=does nothing for a while
=type=command
=env=cli
 !re
=name=do
=summary=executes command
=type=command
=env=cli
... more !re sentences ...
 !re
=name=export
=summary=Print or save an export script that can be used to restore configuration
=type=command
=policy=read,sensitive
 !done
/help
=menu=/system/resource
 !re
=name=cpu
=summary=
=type=menu
 !re
=name=export
=summary=Print or save an export script that can be used to restore configuration
=type=command
=policy=read,sensitive
 !re
=name=get
=summary=Gets value of item's property
=type=command
=policy=read
... more !re sentences ...
 !done
/help
=menu=/interface
 !re
=name=6to4
=summary=
=type=menu
 !re
=name=blink
=summary=
=type=command
... more !re sentences ...
 !done
=description=An Interface is physical or virtual device which provides a connection to an external network.

The "command" argument

If specified, the "command" argument MUST be specified together with a "menu" argument.

The value of the "command" argument is the name of a command, without any menu information attached to it. It is an error if "command" includes "/" or whitespace characters, given that neither of these constitute a valid command name.

When ONLY the "menu" and "command" arguments are both present, the "/help" command should return the available arguments for the specified command, at the specified menu, along with detailed information about the command itself.

The "!done" reply may optionally contain the following attributes:

  • description - A long description of the command, as seen above the argument list, when using "?" after a command.
  • flags - A comma separated list of flags for the command. In its absence, a finite command that doesn't support queries is assumed. Possible values:
    • continious - Indicates that the command will - upon having ONLY its unconditionally required arguments and without being explicitly canceled - run continiously until explicitly canceled or disconnected.
    • queryable - Indicates that the command supports queries. On the command line, this would be the equivalent of a "where" argument being supported in its standard form.

For each argument that the command accepts, a "!re" reply is generated with the following attributes:

  • name - The name of the argument.
  • summary - A short description of the argument, as seen in the help from "?" after a command.
  • flags - Optional. A comma separated list of flags for the argument. If absent, applications should assume an optional named argument with a value. Possible values:
    • unnamed - Indicates that the argument's name may be omitted when using the command from the command line.
    • empty - Indicates that the argument does not have a value, but should instead always be empty.
    • required - Indicates that the argument is required in all calls to the command from any protocol/environment, regardless of the presence or value of any other argument.
    • finite-invert - Indicates that a finite command would become continious in the presence of this argument, or if the command is continious (according to the "!done" reply), it would become finite instead.
  • env - Optional. Works the same as the "env" on a command, but applies only to the particular argument.
  • policy - Optional. A comma separated of additional permissions that are required when this argument is present. Each policy value may also be prefixed with "!" to indicate that an otherwise required permission would not be required when this argument is present.
Icon-note.png

Note: RouterOS should return unnamed parameters in the order in which it would accept them from the command line.


Icon-note.png

Note: The "unnamed" and "empty" flags will allow API clients to emulate the command line more fully, while the "required" flag can be a trivial check if an API client pre-validates commands.


Icon-note.png

Note: Some arguments' availablility over a protocol/environment may be dependent on the presence/absence/value of certain other arguments. The reply must consider the argument as unavailable to a protocol/environment only when it is unconditionally so.


Icon-warn.png

Warning: Applications should not rely on the order of named arguments.


Example:

/help
=menu=/
=command=export
 !re
=name=compact
=summary=
=flags=empty
 !re
=name=file
=summary=File name
=policy=write
 !re
=name=hide-sensetive
=summary=
=flags=empty
=policy=!sensitive
 !re
=name=verbose
=summary=
=flags=empty
 !done
/help
=menu=/tool
=command=ping
 !re
=name=address
=summary=IP address
=flags=unnamed,required
 !re
=name=append
=summary=
=flags=empty
=env=cli
 !re
=name=as-value
=summary=
=flags=empty
=env=cli
 !re
=name=once
=summary=
=flags=empty,finite-invert
... more !re sentences ...
 !done
=flags=continious
/help
=menu=/interface
=command=monitor
 !re
=name=append
=summary=
=flags=empty
=env=cli
 !re
=name=interface
=summary=Interface name
=flags=unnamed,required
 !re
=name=as-value
=summary=
=flags=empty
=env=cli
=name=once
=summary=
=flags=empty,finite-invert
... more !re sentences ...
 !done
=description=Monitor amount of traffic that leaves and enters router through the interface.
=flags=continious
/help
=menu=/interface
=command=print
 !re
=name=append
=summary=
=flags=empty
=env=cli
 !re
=name=as-value
=summary=
=flags=empty
=env=cli
 !re
=name=follow
=summary=
=flags=empty,finite-invert
... more !re sentences ...
 !done
=flags=queryable

The "argument" argument

If present, the "argument" argument MUST be spefied along with both the "menu" and "command" arguments.

The value of the "argument" argument is the name of a command's argument. It is an error if it contains "=" or white spaces, since these do not contitute a valid argument name.

When specified, the "/help" command should return information about possible values for the argument. RouterOS should return both generic and specific information about values. API clients and applications may only partly validate values against the described criterias.

When the argument is empty, there should only be a single "!done" reply, and nothing else.

The "!done" reply may optionally contain the following attributes:

  • description - A long description of the argument, as seen at the top of argument descriptions when using "?" after starting an argument value.
  • flags - A comma separated list of flags for the argument's value. Possible values:
    • negatable - Indicates that the argument's value can also be prefixed with "!" to indicate its negation.

Each criteria is described in a "!re" reply that has at least the attribute "type", which can be one of the following values:

  • datatype - Specifies that the value can be of the specified RouterOS scripting data type.
  • literal - Specifies that the value can be a particular value.
  • enum - Specifies that the value can be one of a set of values.
  • range - Specifies that the value can be a numeric value in a range. Also works for times and IPs from one end to another (after they're converted to integers).
  • network - Specifies that the value can be an IP or IP prefix matching the specified network address.
  • regex - Specifies that the value can be one matching a (POSIX) regular expression.
  • special - Specifies that the value may have certain special semantics that aren't represented by a scripting datatype.
  • lookup - Specifies that the value can be one found at a particular property at a menu.
  • requires - Specifies that another argument is required when this argument is present.

An empty "negate" attribute may optionally be present. If present, indicates that the specified criteria defines values that are NOT allowed.

The attribute "name" may optionally be present. Its value is a comma separated list of names by which other criteria (within the same "/help" call) may refer to that criteria. A named criteria does not apply constraints on the argument's value. If multiple criteria share the same name, a reference to that name may refer to any one of them.

Further attributes can be determined based on the value of "type".

"datatype" criteria

The datatype "!re" reply contains the attribute "value", which contains a comma separated list of scripting data types that the argument may take if present, without regards for any further restrictions or special semantics (e.g. a regular expression is stored as a string, so "str" should be listed).

An argument that may be present, but optionally be empty can use "nothing" as part of that list.

An argument accepting a comma separated list of values should use the "array" datatype (since those are implicitly converted to such lists), alongside the datatypes that a single value may be. If "array" is part of the datatypes, the rest of the criteria should be considered as applying to individual values, as opposed to the array as a whole.

The attribute "maxcount" may also be specified, indicating the maximum number of array elements. In its absence, an array may contain unlimited number of members. In the absence of "array" in the datatypes list, this attribute should be ignored (and ideally, RouterOS should not include it).

The attribute "maxlen" may also be specified, indicating the maximum length of each individual value, if that value is casted to a string.

If multiple such replies are present, the value may match any one of them.

Example:

/help
=menu=/
=command=export
=argument=file
 !re
=type=datatype
=value=str
 !done

Icon-note.png

Note: Applications that support both this and other values of "type" should treat the "datatype" reply as a generic hint, with further replies contraining those types further.


Icon-note.png

Note: Other types complement each other, expanding the total number of allowed values. This is the only type that provides a base that is restricted upon, rather than expanded upon.


"literal" criteria

The literal "!re" reply contains the attribute "value", which contains a single allowed value for the argument. In the presence of multiple "literal" replies, applications should treat any of those values as valid ones.

Icon-warn.png

Warning: The "value" should not be processed in any way. It is NOT a comma separated list or anything else. When a comma separated list is appropriate, the "enum" type can be used instead.


"enum" criteria

The enum "!re" reply contains the "value" attribute, which contains a comma separated list of allowed values.

Example:

/help
=menu=/ip/firewall/mangle
=command=add
=argument=chain
 !re
=type=datatype
=value=str
 !re
=type=enum
=value=prerouting,input,forward,output,postrouting
 !done

"range" criteria

The range "!re" reply contains two attributes - "from" and "to", which contain respectively the starting/minimum (inclusive) and final/maximum (inclusive) allowed values for the range. If multiple range replies are present, the list of allowed values should include the union of all ranges.

The value of the "from" and "to" attributes may be any discrete type other than "bool" and "id" (i.e. "num", "time" and "ip"), and is applied only over those types.

If the argument is said to accept other datatypes (via a "datatype" reply), this constraint does not apply to values matching those other data types.

Example:

/help
=menu=/queue/simple
=command=add
=argument=priority
 !re
=type=datatype
=value=num
 !re
=type=range
=from=1
=to=8
 !done

"network" criteria

The network "!re" reply contains the attribute "value", the value of which is an IP prefix (either IPv4 or IPv6) that the argument may match. If multiple such replies are present, allowed values include their union.

Applications should constrain the value according to this criteria if the argument can be casted to an IP address (either IPv4 or IPv6, depending on "value"'s value) and back without a loss. This includes scripting datatypes "ip", "num" and "time".

If the argument is said to accept other datatypes (via a "datatype" reply), this constraint does not apply to values matching those other datatypes.

"regex" criteria

The regex "!re" reply contains the attribute "value", the value of which is a POSIX regular expression that the argument may match. If multiple such replies are present, the argument may match any of the regular expressions.

This constraint applies only to string values. If other datatypes are allowed (via a "datatype" reply), applications should treat a value as allowed if it can be casted to those other types, even if this criteria is not matched.

"special" criteria

The special "!re" reply contains the attribute "value", the value of which can be one of:

  • regex - Indicates that the value will be used as a POSIX regular expression. Applies only when the argument value is a string.
  • range - Indicates that the value is a range, specified as either a single numeric value, or as "Start-End" value. The values for the start and end are all valid single numeric values defined by other criteria.
  • script - Indicates that the value is a source of a RouterOS script. Applies only when the argument value is a string.

Example:

/help
=menu=/ip/firewall/nat
=command=add
=argument=packet-size
 !re
=type=datatype
=value=num
 !re
=type=range
=from=0
=to=65535
 !re
=type=special
=value=range
 !done
=flags=negatable
/help
=menu=/ip/firewall/layer7
=command=add
=argument=regexp
 !re
=type=datatype
=value=str
 !re
=type=special
=value=regex
 !done
/help
=menu=/system/script
=command=add
=argument=source
 !re
=type=datatype
=value=str
 !re
=type=special
=value=script
 !done

"lookup" criteria

The lookup "!re" reply contains the attributes "menu" and "proplist", which specify respectively a menu (using API syntax) and a comma separated list of properties, the values of which are allowed values for the argument.

The attribute "menu" may be omitted when it refers to the current menu.

The values implied by this criteria constrain only datatypes of the referenced properties. So for example, if a value can be of types "ip" or "str", and the properties from the "lookup" can only be strings, valid values are any "ip" value, or a str value from the "lookup" values. On the other hand, if the referenced properties can be of types "ip" or "str" as well, then valid values are only the referenced values.

Icon-warn.png

Warning: The attribute "proplist" is different from the common ".proplist" attribute.


Example:

/help
=menu=/interface
=command=monitor
=argument=interface
 !re
=type=datatype
=value=id,str,array
=maxcount=100
 !re
=type=lookup
=proplist=.id,name
 !done
/help
=menu=/queue/simple
=command=add
=argument=target
 !re
=type=datatype
=value=ip-prefix,ip6-prefix,str,array
=maxcount=128
 !re
=type=lookup
=menu=/interface
=proplist=name
 !done

"requires" criteria

The requires "!re" reply contains the attribute "argument", the value of which is the name of another argument from the current command. When a call to the command is made that includes the current argument, the referenced argument must also be specified.

The attribute "value" may optionally be specified to indicate that the referenced argument must match particular values for the current argument to be allowed. The value of the "value" attribute is a comma separated list of names of criterias.

The attribute "if-value" may optionally be specified to indicate that the referenced argument is required only if the current argument is given particular values. The value of the "if-value" attribute is a comma separated list of names of criterias.

When "value" and "if-value" are used together, it means that the referenced argument must have a value matching "value" criterias IF the current argument has a value matching the "if-value" criterias.

Example:

/help
=menu=/ip/firewall/mangle
=command=add
=argument=port
 !re
=type=datatype
=value=num,array
=maxcount=16
 !re
=type=range
=from=0
=to=65535
 !re
=type=special
=value=range
 !re
=type=enum
=name=protocolsWithPorts
=value=tcp,udp
 !re
=type=requires
=argument=protocol
=value=protocolsWithPorts
 !done
=flags=negatable

Security considerations

This command can expose useful information to potential attackers. On the other hand, a potential attacker may gather most of that same information from other sources (e.g. the manual) by merely knowing the RouterOS version and the list of available packages.

As such, this command requires the same level of protection that RouterOS has for revealing its version and package list over the API protocol. Namely:

  • The client must first log in successfully - this command is not available to "guests".
  • The logged in user needs to have "read" permissions. The "read" permission is required for a client to successfully execute "/system/resource/print" and "/system/package/print" anyway.

Alternatively, there may be an anonymous version of this command, which reports only the anonymously available commands ("/login" and "/quit"), and their details, and has an empty !done response when called with no arguments.

Misc

Author: boen_robot.

Feedback: forum topic.

Originally proposed in: forum topic.