The PyData Sphinx Theme#
This is a simple, Bootstrap-based Sphinx theme from the PyData community. This site is a guide for using the theme, and a demo for how it looks with various elements.
Other sites that are using this theme:
Binder: https://docs.mybinder.org/
JupyterLab: https://jupyterlab.readthedocs.io/en/latest/
JupyterHub: https://jupyterhub.readthedocs.io/en/latest/
Matplotlib: https://matplotlib.org/stable/
Pandas: https://pandas.pydata.org/docs/
JupyterHub and Binder: https://docs.mybinder.org/, http://z2jh.jupyter.org/en/latest/, https://repo2docker.readthedocs.io/en/latest/, https://jupyterhub-team-compass.readthedocs.io/en/latest/
Jupyter Book beta version uses an extension of this theme: https://beta.jupyterbook.org
Fairlearn: https://fairlearn.org/main/about/
PyVista: https://docs.pyvista.org
User Guide#
Installation#
The theme is available on PyPI and conda-forge, and can thus be installed with:
$ pip install pydata-sphinx-theme
$ conda install pydata-sphinx-theme --channel conda-forge
Then, in the conf.py
of your sphinx docs, you update the html_theme
configuration option:
html_theme = "pydata_sphinx_theme"
If you want to track the development version of the theme, you can install it from the git repo:
$ pip install git+https://github.com/pydata/pydata-sphinx-theme.git@master
or in a conda environment yml file, you can add:
- pip:
- git+https://github.com/pydata/pydata-sphinx-theme.git@master
Configuration#
There are a number of options for configuring your site’s look and feel.
All configuration options are passed with the html_theme_options
variable
in your conf.py
file. This is a dictionary with key: val
pairs that
you can configure in various ways. This page describes the options available to you.
Configure project logo#
To add a logo that’s placed at the left of your nav bar, put a logo file under your doc path’s _static folder, and use the following configuration:
html_logo = "_static/logo.png"
The logo links to master_doc
(usually the first page of your documentation) by default.
If you’d like it to link to another page or use an external link instead, use the following configuration:
html_theme_options = {
"logo_link": "<other page or external link>"
}
Configure icon links#
You can add icon links to show up to the right of your main navigation bar.
These links take the following form:
html_theme_options = {
...
"icon_links": [
{
# Label for this link
"name": "GitHub",
# URL where the link will redirect
"url": "https://github.com/<your-org>/<your-repo>", # required
# Icon class (if "type": "fontawesome"), or path to local image (if "type": "local")
"icon": "fab fa-github-square",
# Whether icon should be a FontAwesome class, or a local file
"type": "fontawesome OR local", # Default is fontawesome
}
]
}
There are two kinds of icons you can use, described below:
FontAwesome icons#
FontAwesome is a collection of icons that are commonly used in websites. They include both generic shape icons (e.g., “arrow-down”), as well as brand-specific icons (e.g. “github”).
You can use FontAwesome icons by specifying "type": "fontawesome"
, and
specifying a FontAwesome class in the icon
value.
The value of icon
can be any full
FontAwesome 5 Free icon.
In addition to the main icon class, e.g. fa-cat
, the “style” class must
also be provided e.g. fab for branding, or fas for solid.
Here are several examples:
html_theme_options = {
...
"icon_links": [
{
"name": "GitHub",
"url": "https://github.com/<your-org>/<your-repo>",
"icon": "fab fa-github-square",
"type": "fontawesome",
},
{
"name": "GitLab",
"url": "https://gitlab.com/<your-org>/<your-repo>",
"icon": "fab fa-gitlab",
"type": "fontawesome",
},
{
"name": "Twitter",
"url": "https://twitter.com/<your-handle>",
"icon": "fab fa-twitter-square",
# The default for `type` is `fontawesome` so it is not actually required in any of the above examples as it is shown here
},
],
...
}
Hint
To get custom colors like “Twitter blue”, use the following in your CSS,
e.g. custom.css
:
i.fa-twitter-square:before {
color: #55acee;
}
This has already been added for the brands that have shortcuts.
Local image icons#
If you’d like to display an icon image that is not in the FontAwesome icons library,
you may instead specify a path to a local image that will be used for the icon.
To do so, use "type": "local"
, and add a path to an image
relative to your documentation root in the icon
value.
Here is an example:
html_theme_options = {
...
"icon_links": [
{
"name": "PyData",
"url": "https://pydata.org",
"icon": "_static/pydata-logo-square.png",
"type": "local",
},
],
...
}
Tip
Use .svg
images for a higher-resolution output that behaves similarly across screen sizes.
Icon Link Shortcuts#
There are a few shortcuts supported to minimize configuration for commonly-used services.
These may be removed in a future release in favor of icon_links
:
html_theme_options = {
...
"github_url": "https://github.com/<your-org>/<your-repo>",
"gitlab_url": "https://gitlab.com/<your-org>/<your-repo>",
"bitbucket_url": "https://bitbucket.org/<your-org>/<your-repo>",
"twitter_url": "https://twitter.com/<your-handle>",
...
}
Additionally, the screen-reader accessible label for this menu can be configured:
html_theme_options = {
...
"icon_links_label": "Quick Links",
...
}
Adding favicons#
pydata_sphinx_theme
supports the
standard sphinx favicon configuration,
using html_favicon
.
Additionally, pydata_sphinx_theme
allows you to add any number of
browser- or device-specific favicons of any size. To define arbitrary favicons,
use the favicons
configuration key. The href
value can be either an
absolute URL (beginning with http
) or a local path relative to your
html_static_path
:
html_theme_options = {
"favicons": [
{
"rel": "icon",
"sizes": "16x16",
"href": "https://secure.example.com/favicon/favicon-16x16.png",
},
{
"rel": "icon",
"sizes": "32x32",
"href": "favicon-32x32.png",
},
{
"rel": "apple-touch-icon",
"sizes": "180x180",
"href": "apple-touch-icon-180x180.png"
},
]
}
pydata_sphinx_theme
will add link
tags to your document’s head
section, following this pattern:
<link rel="{{ favicon.rel }}" sizes="{{ favicon.sizes }}" href="{{ favicon.href }}">
Add a dropdown to switch between docs versions#
You can add a button to your site that allows users to
switch between versions of your documentation. The links in the version
switcher will differ depending on which page of the docs is being viewed. For
example, on the page https://mysite.org/en/v2.0/changelog.html
, the
switcher links will go to changelog.html
in the other versions of your
docs. When clicked, the switcher will check for the existence of that page, and
if it doesn’t exist, redirect to the homepage of that docs version instead.
The switcher requires the following configuration steps:
Add a JSON file containing a list of the documentation versions that the switcher should show on each page.
Add a configuration dictionary called
switcher
to thehtml_theme_options
dict inconf.py
.switcher
should have 3 keys:json_url
: the persistent location of the JSON file described above.version_match
: a string stating the version of the documentation that is currently being browsed.
Specify where to place the switcher in your page layout. For example, add the
"version-switcher"
template to one of the layout lists inhtml_theme_options
(e.g.,navbar_end
,footer_items
, etc).
Below is a more in-depth description of each of these configuration steps.
Add a JSON file to define your switcher’s versions#
First, write a JSON file stating which versions of your docs will be listed in the switcher’s dropdown menu. That file should contain a list of entries that each can have the following fields:
version
: a version string. This is checked againstswitcher['version_match']
to provide styling to the switcher.url
: the URL for this version.name
: an optional name to display in the switcher dropdown instead of the version string (e.g., “latest”, “stable”, “dev”, etc).
Here is an example JSON file:
[
{
"name": "v2.1 (stable)",
"version": "2.1",
"url": "https://mysite.org/en/2.1/index.html"
},
{
"version": "2.1rc1",
"url": "https://mysite.org/en/2.1rc1/index.html"
},
{
"version": "2.0",
"url": "https://mysite.org/en/2.0/index.html"
},
{
"version": "1.0",
"url": "https://mysite.org/en/1.0/index.html"
}
]
See the discussion of switcher['json_url']
(below) for options of where to
save the JSON file.
Configure switcher['json_url']
#
The JSON file needs to be at a stable, persistent, fully-resolved URL (i.e., not specified as a path relative to the sphinx root of the current doc build). Each version of your documentation should point to the same URL, so that as new versions are added to the JSON file all the older versions of the docs will gain switcher dropdown entries linking to the new versions. This could be done a few different ways:
The location could be one that is always associated with the most recent documentation build (i.e., if your docs server aliases “latest” to the most recent version, it could point to a location in the build tree of version “latest”). For example:
html_theme_options = { ..., "switcher": { "json_url": "https://mysite.org/en/latest/_static/switcher.json", } }
In this case the JSON is versioned alongside the rest of the docs pages but only the most recent version is ever loaded (even by older versions of the docs).
The JSON could be saved in a folder that is listed under your site’s
html_static_path
configuration. See the Sphinx static path documentation for more information.The JSON could be stored outside the doc build trees. This probably means it would be outside the software repo, and would require you to add new version entries to the JSON file manually as part of your release process. Example:
html_theme_options = { ..., "switcher": { "json_url": "https://mysite.org/switcher.json", } }
Configure switcher['version_match']
#
This configuration value tells the switcher what docs version is currently being viewed, and is used to style the switcher (i.e., to highlight the current docs version in the switcher’s dropdown menu, and to change the text displayed on the switcher button).
Typically you can re-use one of the sphinx variables version
or release
as the value of switcher['version_match']
; which one you use
depends on how granular your docs versioning is. See
the Sphinx “project info” documentation
for more information). Example:
version = my_package_name.__version__.replace("dev0", "") # may differ
html_theme_options = {
...,
"switcher": {
"version_match": version,
}
}
Specify where to display the switcher#
Finally, tell the theme where on your site’s pages you want the switcher to
appear. There are many choices here: you can add "version-switcher"
to one
of the locations in html_theme_options
(e.g., navbar_end
,
footer_items
, etc). For example:
html_theme_options = {
...,
"navbar_end": ["version-switcher"]
}
Alternatively, you could override one of the other templates to include the
version switcher in a sidebar. For example, you could define
_templates/sidebar-nav-bs.html
as:
{%- include 'version-switcher.html' -%}
{{ super() }}
to insert a version switcher at the top of the left sidebar, while still keeping the default navigation below it. See Add/Remove items from theme sections for more information.
Configure the search bar position#
To modify the position of the search bar, add the search-field.html
template to your sidebar, or to one of the navbar positions, depending
on where you want it to be placed.
For example, if you’d like the search field to be in your side-bar, add it to the sidebar templates like so:
html_sidebars = {
"**": ["search-field.html", "sidebar-nav-bs.html", "sidebar-ethical-ads.html"]
}
If instead you’d like to put the search bar in the top navbar, use the following configuration:
html_theme_options = {
"navbar_end": ["navbar-icon-links.html", "search-field.html"]
}
Note
By default the search bar is placed in the sidebar. If you wish to move it to the navbar, explicitly define a list of sidebar templates in html_sidebars and omit the search-field.html entry.
Configure the search bar text#
To modify the text that is in the search bar before people click on it, add the
following configuration to your conf.py
file:
html_theme_options = {
"search_bar_text": "Your text here..."
}
Google Analytics#
If the google_analytics_id
config option is specified (like UA-XXXXXXX
),
Google Analytics’ javascript is included in the html pages.
html_theme_options = {
"google_analytics_id": "UA-XXXXXXX",
}
Changing pages with keyboard presses#
By default, pydata-sphinx-theme
allows users to move to the previous/next
page using the left/right arrow keys on a keyboard. To disable this behavior,
use the following configuration:
html_theme_options = {
"navigation_with_keys": False
}
Show more levels of the in-page TOC by default#
Normally only the 2nd-level headers of a page are show in the right table of contents, and deeper levels are only shown when they are part of an active section (when it is scrolled on screen).
You can show deeper levels by default by using the following configuration, indicating how many levels should be displayed:
html_theme_options = {
"show_toc_level": 2
}
All headings up to and including the level specified will now be shown regardless of what is displayed on the page.
Improve build speed and performance#
By default this theme includes all of your documentation links in a collapsible sidebar.
However, this may slow down your documentation builds considerably if you have
a lot of documentation pages. This is most common with documentation for projects
with a large API, which use the .. autosummary::
directive to generate
API documentation.
To improve the performance of your builds in these cases, see Navigation depth and collapsing the sidebar.
Add/Remove items from theme sections#
There are a few major theme sections that you can customize to add/remove components, or add your own components. Each section is configured with a list of html templates - these are snippets of HTML that are inserted into the section by Sphinx.
You can choose which templates show up in each section, as well as the order in which they appear. This page describes the major areas that you can customize.
Note
When configuring templates in each section, you may omit the .html
suffix after each template if you wish.
A list of built-in templates you can insert into sections#
Below is a list of built-in templates that you can insert into any section. Note that some of them may have CSS rules that assume a specific section (and will be named accordingly).
icon-links.html
search-field.html
copyright.html
edit-this-page.html
last-updated.html
navbar-icon-links.html
navbar-logo.html
navbar-nav.html
page-toc.html
sidebar-ethical-ads.html
sidebar-nav-bs.html
sphinx-version.html
version-switcher.html
Add your own HTML templates to theme sections#
If you’d like to add your own custom template to any of these sections, you could do so with the following steps:
Create an HTML file in a folder called
_templates
. For example, if you wanted to display the version of your documentation using a Jinja template, you could create a file:_templates/version.html
and put the following in it:<!-- This will display the version of the docs --> {{ version }}
Now add the file to your menu items for one of the sections above. For example:
html_theme_options = { ... "navbar_start": ["navbar-logo", "version"], ... }
Customizing the theme#
In addition to the configuration options detailed at Configuration, it is also possible to customize the HTML layout and CSS style of the theme.
Custom CSS Stylesheets#
You may customize the theme’s CSS by creating a custom stylesheet that Sphinx uses to build your site. Any rules in this style-sheet will over-ride the default theme rules.
To add a custom stylesheet, follow these steps:
Create a CSS stylesheet in
_static/css/custom.css
, and add the CSS rules you wish.Attach the stylesheet to your Sphinx build. Add the following to
conf.py
html_static_path = ['_static'] html_css_files = [ 'css/custom.css', ]
When you build your documentation, this stylesheet should now be activated.
CSS Theme variables#
This theme defines several CSS variables that can be used to quickly control behavior across your documentation.
These are based on top of the basic Bootstrap CSS variables extended with some theme specific variables. An overview of all variables and every default is defined in the pydata default CSS variables file.
In order to change a variable, follow these steps:
Add a custom CSS stylesheet. This is where we’ll configure the variables.
Underneath a
:root
section, add the variables you wish to update. For example, to update the base font size, you might add this tocustom.css
::root { --pst-font-size-base: 17px; }
Important
Note that these are CSS variables and not SASS variables. The theme is defined with CSS variables, not SASS variables!
For a complete list of the theme variables that you may override, see the theme variables defaults CSS file:
/* Provided by Sphinx's 'basic' theme, and included in the final set of assets */
@import "../basic.css";
:root {
/*****************************************************************************
* Theme config
**/
--pst-header-height: 60px;
/*****************************************************************************
* Font size
**/
--pst-font-size-base: 15px; /* base font size - applied at body / html level */
/* heading font sizes */
--pst-font-size-h1: 36px;
--pst-font-size-h2: 32px;
--pst-font-size-h3: 26px;
--pst-font-size-h4: 21px;
--pst-font-size-h5: 18px;
--pst-font-size-h6: 16px;
/* smaller then heading font sizes*/
--pst-font-size-milli: 12px;
--pst-sidebar-font-size: 0.9em;
--pst-sidebar-caption-font-size: 0.9em;
/*****************************************************************************
* Font family
**/
/* These are adapted from https://systemfontstack.com/ */
--pst-font-family-base-system: -apple-system, BlinkMacSystemFont, Segoe UI,
"Helvetica Neue", Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji,
Segoe UI Symbol;
--pst-font-family-monospace-system: "SFMono-Regular", Menlo, Consolas, Monaco,
Liberation Mono, Lucida Console, monospace;
--pst-font-family-base: var(--pst-font-family-base-system);
--pst-font-family-heading: var(--pst-font-family-base);
--pst-font-family-monospace: var(--pst-font-family-monospace-system);
/*****************************************************************************
* Color
*
* Colors are defined in rgb string way, "red, green, blue"
**/
--pst-color-primary: 19, 6, 84;
--pst-color-success: 40, 167, 69;
--pst-color-info: 0, 123, 255; /*23, 162, 184;*/
--pst-color-warning: 255, 193, 7;
--pst-color-danger: 220, 53, 69;
--pst-color-text-base: 51, 51, 51;
--pst-color-h1: var(--pst-color-primary);
--pst-color-h2: var(--pst-color-primary);
--pst-color-h3: var(--pst-color-text-base);
--pst-color-h4: var(--pst-color-text-base);
--pst-color-h5: var(--pst-color-text-base);
--pst-color-h6: var(--pst-color-text-base);
--pst-color-paragraph: var(--pst-color-text-base);
--pst-color-link: 0, 91, 129;
--pst-color-link-hover: 227, 46, 0;
--pst-color-headerlink: 198, 15, 15;
--pst-color-headerlink-hover: 255, 255, 255;
--pst-color-preformatted-text: 34, 34, 34;
--pst-color-preformatted-background: 250, 250, 250;
--pst-color-inline-code: 232, 62, 140;
--pst-color-active-navigation: 19, 6, 84;
--pst-color-navbar-link: 77, 77, 77;
--pst-color-navbar-link-hover: var(--pst-color-active-navigation);
--pst-color-navbar-link-active: var(--pst-color-active-navigation);
--pst-color-sidebar-link: 77, 77, 77;
--pst-color-sidebar-link-hover: var(--pst-color-active-navigation);
--pst-color-sidebar-link-active: var(--pst-color-active-navigation);
--pst-color-sidebar-expander-background-hover: 244, 244, 244;
--pst-color-sidebar-caption: 77, 77, 77;
--pst-color-toc-link: 119, 117, 122;
--pst-color-toc-link-hover: var(--pst-color-active-navigation);
--pst-color-toc-link-active: var(--pst-color-active-navigation);
/*****************************************************************************
* Icon
**/
/* font awesome icons*/
--pst-icon-check-circle: "\f058";
--pst-icon-info-circle: "\f05a";
--pst-icon-exclamation-triangle: "\f071";
--pst-icon-exclamation-circle: "\f06a";
--pst-icon-times-circle: "\f057";
--pst-icon-lightbulb: "\f0eb";
/*****************************************************************************
* Admonitions
**/
--pst-color-admonition-default: var(--pst-color-info);
--pst-color-admonition-note: var(--pst-color-info);
--pst-color-admonition-attention: var(--pst-color-warning);
--pst-color-admonition-caution: var(--pst-color-warning);
--pst-color-admonition-warning: var(--pst-color-warning);
--pst-color-admonition-danger: var(--pst-color-danger);
--pst-color-admonition-error: var(--pst-color-danger);
--pst-color-admonition-hint: var(--pst-color-success);
--pst-color-admonition-tip: var(--pst-color-success);
--pst-color-admonition-important: var(--pst-color-success);
--pst-icon-admonition-default: var(--pst-icon-info-circle);
--pst-icon-admonition-note: var(--pst-icon-info-circle);
--pst-icon-admonition-attention: var(--pst-icon-exclamation-circle);
--pst-icon-admonition-caution: var(--pst-icon-exclamation-triangle);
--pst-icon-admonition-warning: var(--pst-icon-exclamation-triangle);
--pst-icon-admonition-danger: var(--pst-icon-exclamation-triangle);
--pst-icon-admonition-error: var(--pst-icon-times-circle);
--pst-icon-admonition-hint: var(--pst-icon-lightbulb);
--pst-icon-admonition-tip: var(--pst-icon-lightbulb);
--pst-icon-admonition-important: var(--pst-icon-exclamation-circle);
/*****************************************************************************
* versionmodified
**/
--pst-color-versionmodified-default: var(--pst-color-info);
--pst-color-versionmodified-added: var(--pst-color-success);
--pst-color-versionmodified-changed: var(--pst-color-warning);
--pst-color-versionmodified-deprecated: var(--pst-color-danger);
--pst-icon-versionmodified-default: var(--pst-icon-exclamation-circle);
--pst-icon-versionmodified-added: var(--pst-icon-exclamation-circle);
--pst-icon-versionmodified-changed: var(--pst-icon-exclamation-circle);
--pst-icon-versionmodified-deprecated: var(--pst-icon-exclamation-circle);
}
Replacing/Removing Fonts#
The theme includes the FontAwesome 5 Free
icon font (the .fa, .far, .fas
styles, which are used for
icon links and admonitions).
This is the only vendored font, and otherwise the theme by default relies on
available system fonts for normal body text and headers.
Attention
Previously-included fonts like Lato have been removed, preferring the most common default system fonts of the reader’s computer. This provides both better performance, and better script/glyph coverage than custom fonts, and is recommended in most cases.
The default body and header fonts can be changed as follows:
Using Custom CSS Stylesheets, you can specify which fonts to use for body, header and monospace text. For example, the following can be added to a custom css file:
:root { --pst-font-family-base: Verdana, var(--pst-font-family-base-system); --pst-font-family-heading: Cambria, Georgia, Times, var(--pst-font-family-base-system); --pst-font-family-monospace: Courier, var(--pst-font-family-monospace-system); }
The
-system
variables are available to use as fallback to the default fonts.If the font you want to specify in the section above is not generally available by default, you will additionally need to ensure the font is loaded. For example, you could download and vendor the font in the
_static
directory of your Sphinx site, and then update the base template to load the font resources:Configure the template_path in your
conf.py
Create a custom
layout.html
Jinja2 template which overloads thefonts
block (example for loading the Lato font that is included in the_static/vendor
directory):{% extends "pydata_sphinx_theme/layout.html" %} {% block fonts %} <!-- add `style` or `link` tags with your CSS `@font-face` declarations here --> <!-- ... and optionally preload the `woff2` for snappier page loads --> <link rel="stylesheet" href="{{ pathto('_static/vendor/lato_latin-ext/1.44.1/index.css', 1) }}"> {% endblock %}
To reduce the Flash of Unstyled Content, you may wish to explore various options for preloading content, specifically the binary font files. This ensure the files will be loaded before waiting for the CSS to be parsed, but should be used with care.
Accessibility#
Creating and publishing content that does not exclude audiences with limited abilities of various kinds is challenging, but also important, to achieve and then maintain.
While there is no one-size-fits-all solution to maintaining accessible content, this theme and documentation site use some techniques to avoid common content shortcomings.
Note
Issues and pull requests to identify or fix accessibility issues on this theme or site are heartily welcomed!
In Configuration#
Some minor configuration options in a site’s conf.py
can impact the
accessibility of content generated by this theme, and Sphinx in general.
Natural Language#
If not using a more robust internationalization approach , specifying at least the baseline natural language will help assistive technology identify if the content is in a language the reader understands.
Hint
Specifying a language
will propagate to the top-level html tag.
language = "en"
Site Map#
Site maps, usually served from a file called sitemap.xml are a broadly-employed approach to telling programs like search engines and assistive technologies where different content appears on a website.
If using a service like ReadTheDocs, these files will be created for you automatically, but for some of the other approaches below, it’s handy to generate a sitemap.xml locally or in CI with a tool like sphinx-sitemap.
Hint
For a simple site (no extra languages or versions), ensure sphinx-sitemap
is installed in your documentation environment, and modify your conf.py
:
extensions += ["sphinx_sitemap"]
html_baseurl = os.environ.get("SPHINX_HTML_BASE_URL", "http://127.0.0.1:8000/")
sitemap_locales = [None]
sitemap_url_scheme = "{link}"
In Your Source#
Note
Stay tuned for more ideas here as we learn more working on this site!
In the Browser#
A number of in-browser tools exist for interactively debugging the accessibility of a single page at a time, and can be useful during the content development cycle.
Built-in tools#
Most major browsers, including Firefox and Chrome include significant accessibility tooling in their development experience. Exploring these, and the modes they offer, can help to quickly pinpoint issues, and often include links to standards.
tota11y#
tota11y is an open source “bookmarklet” which modifies the currently-loaded page in-place, and highlights a number of accessibility issues.
WAVE#
WAVE is a proprietary (but gratis) browser extension which can highlight a large number of issues.
In Continuous Integration#
A number of automated tools are available for assessing glaring accessibility issues across a number of pages at once, usually with many configurable options.
Lighthouse#
Lighthouse, which provides automated assessment of basic accessibility issues in addition to search engine automation, page performance, and other best practices.
Hint
Specifically, foo-software/lighthouse-check-action is run on selected pages from the generated documentation site.
Pa11y CI#
Pa11y CI is a command line tool which can check a number of accessibility standards. It is most effective when paired with a sitemap.xml, discussed above.
Hint
This approach is more involved: for this site, we’ve written some custom runners which:
start a static file server locally with the docs site
run pa11y-ci against the site’s sitemap.xml
read known failures in a a11y-roadmap.txt file
generate HTML reports (including all errors)
perform some light parsing to generate some short reports
archive the reports in CI
Contribute#
These pages contain information about how you can get up-and-running with a development version of this theme, and how you can contribute to the project.
Workflow for contributing changes#
We follow a typical GitHub workflow of:
create a personal fork of this repo
create a branch
open a pull request
fix findings of various linters and checks
work through code review
For each pull request, the demo site is built and deployed to make it easier to review the changes in the PR. To access this, click on the “ReadTheDocs” preview in the CI/CD jobs.
Location and structure of documentation#
The documentation for this theme is in the docs/
folder.
It is structured as a Sphinx documentation site.
The content is written in a combination of reStructuredText and MyST Markdown.
Location and structure of CSS/JS assets#
The CSS and JS for this theme are built for the browser from src/pydata_sphinx_theme/assets/*
with
webpack. The main entrypoints are:
CSS:
src/pydata_sphinx_theme/assets/styles/index.scss
JS:
src/pydata_sphinx_theme/assets/scripts/index.js
provides add-on Bootstrap features, as well as some custom navigation behavior
webpack:
webpack.config.js
captures the techniques for transforming the JS and CSS source files in
src/pydata_sphinx_theme/assets/*
into the production assets insrc/theme/pydata_sphinx_theme/static/
For more information about developing this theme, see the sections below and in the left sidebar.
Get started with development#
This section covers the simplest way to get started developing this theme locally so that you can contribute. It uses automation and as few steps as possible to get things done. If you’d like to do more operations manually, see Set up a manual development environment.
Clone the repository#
First off you’ll need your own copy of the pydata-sphinx-theme
codebase.
You can clone it for local development like so:
Fork the repository so you have your own copy on GitHub. See the GitHub forking guide for more information.
Clone the repository locally so that you have a local copy to work from:
$ git clone https://github.com/{{ YOUR USERNAME }}/pydata-sphinx-theme $ cd pydata-sphinx-theme
Install your tools#
Building a Sphinx site uses a combination of Python and Jinja to manage HTML, SCSS, and Javascript. To simplify this process, we use a few helper tools:
The Sphinx Theme Builder to automatically perform compilation of web assets.
pre-commit for automatically enforcing code standards and quality checks before commits.
nox, for automating common development tasks.
In particular, nox
can be used to automatically create isolated local development environments with all of the correct packages installed to work on the theme.
The rest of this guide focuses on using nox
to start with a basic environment.
See also
The information on this page covers the basics to get you started, for information about manually compiling assets, see Set up a manual development environment.
Setup nox
#
To start, install nox
:
$ pip install nox
You can call nox
from the command line in order to perform common actions that are needed in building the theme.
nox
operates with isolated environments, so each action has its own packages installed in a local directory (.nox
).
For common development actions, you’ll simply need to use nox
and won’t need to set up any other packages.
Setup pre-commit
#
pre-commit
allows us to run several checks on the codebase every time a new Git commit is made.
This ensures standards and basic quality control for our code.
Install pre-commit
with the following command:
$ pip install pre-commit
then navigate to this repository’s folder and activate it like so:
$ pre-commit install
This will install the necessary dependencies to run pre-commit
every time you make a commit with Git.
Note
Your pre-commit
dependencies will be installed in the environment from which you’re calling pre-commit
, nox
, etc.
They will not be installed in the isolated environments used by nox
.
Build the documentation#
Now that you have nox
installed and cloned the repository, you should be able to build the documentation locally.
To build the documentation with nox
, run the following command:
$ nox -s docs
This will install the necessary dependencies and build the documentation located in the docs/
folder.
They will be placed in a docs/_build/html
folder.
If the docs have already been built, it will only build new pages that have been updated.
You can open one of the HTML files there to preview the documentation locally.
Alternatively, you can invoke the built-in Python http.server with:
$ python -m http.server -d docs/_build/html/
This will print a local URL that you can open in a browser to explore the HTML files.
Change content and re-build#
Now that you’ve built the documentation, edit one of the source files to see how the documentation updates with new builds.
Make an edit to a page. For example, add a word or fix a typo on any page.
Rebuild the documentation with
nox -s docs
It should go much faster this time, because nox
is re-using the old environment, and because Sphinx has cached the pages that you didn’t change.
Compile the CSS/JS assets#
The source files for CSS and JS assets are in src/pydata_sphinx_theme/assets
.
These are then built and bundled with the theme (e.g., scss
is turned into css
).
To compile the CSS/JS assets with nox
, run the following command:
$ nox -s compile
This will compile all assets and place them in the appropriate folder to be used with documentation builds.
Note
Compiled assets are not committed to git.
The sphinx-theme-builder
will bundle these assets automatically when we make a new release, but we do not manually commit these compiled assets to git history.
Run a development server#
You can combine the above two actions and run a development server so that changes to src/
are automatically bundled with the package, and the documentation is immediately reloaded in a live preview window.
To run the development server with nox
, run the following command:
$ nox -s docs-live
When working on the theme, saving changes to any of these directories:
src/js/index.js
src/scss/index.scss
docs/**/*.rst
docs/**/*.py
will cause the development server to do the following:
bundle/copy the CSS, JS, and vendored fonts
regenerate the Jinja2 macros
re-run Sphinx
Run the tests#
This theme uses pytest
for its testing, with a lightweight fixture defined
in the test_build.py
script that makes it easy to run a Sphinx build using
this theme and inspect the results.
In addition, we use pytest-regressions
to ensure that the HTML generated by the theme is what we’d expect. This module
provides a file_regression
fixture that will check the contents of an object
against a reference file on disk. If the structure of the two differs, then the
test will fail. If we expect the structure to differ, then delete the file on
disk and run the test. A new file will be created, and subsequent tests will pass.
To run the tests with nox
, run the following command:
$ nox -s test
Topic guides and how-tos#
These sections cover common operations and topics that are relevant to developing this theme.
Make a release#
This theme uses GitHub tags and releases to automatically push new releases to PyPI. For information on this process, see the release checklist.
Update JavaScript dependencies and their versions#
The javascript dependencies for this package are defined in package.json
, and broken down into a few categories like dependencies
and devDependencies
.
To update or add JS dependency, modify (or append to) the list of packages that are listed in each of these sections.
The next time you build the documentation (either with nox
or with stb
), these new dependencies will be installed and bundled with the theme.
Using nox
#
Here are a few extra tips for using nox
.
See also
The nox
command line documentation has a lot of helpful tips for extra functionality you can enable with the CLI.
Re-install dependencies#
To re-execute the installation commands, use this pattern:
$ nox -s docs -- reinstall
Or to completely remove the environment generated by nox
and start from scratch:
$ rm -rf .nox/docs
Use nox
with your global environment#
If you’d like to use nox
with your global environment (the one from which you are calling nox
), you can do so with:
$ nox --force-venv-backend none
# alternatively:
$ nox --no-venv
Using none
will re-use your current global environment.
See
the nox documentation for more details.
Using pre-commit
#
Here are a few tips for using pre-commit
:
Skip the pre-commit checks#
Run the following command:
$ git commit --no-verify
Run pre-commit on all files#
By default, pre-commit
will run its checks on files that have been modified in a commit.
To instead run it on all files, use this command:
$ pre-commit run --all-files
# Alternatively
$ pre-commit run -a
Web assets (CSS/JS/Fonts)#
This theme includes several web assets to ease development and design.
The configuration for our asset compilation is in webpack.config.js
.
Compile and bundle assets#
When assets are compiled, static versions are placed in various places in the theme’s static folder:
src/pydata_sphinx_theme/theme/pydata_sphinx_theme/static
For many assets, a <hash>
is generated and appended to the end of its reference in the HTML templates of the theme.
This ensures the correct asset versions are served when viewers return to your
site after upgrading the theme.
To compile the assets and bundle them with the theme, run this command:
$ nox -s compile
Styles (SCSS) and Scripts (JS)#
There are two relevant places for CSS/JS assets:
src/pydata_sphinx_theme/assets/styles
has source files for SCSS assets. These will be compiled to CSS.src/pydata_sphinx_theme/assets/scripts
has source files for JS assets. These will be compiled to JS and import several vendored libraries (like Bootstrap).src/pydata_sphinx_theme/theme/pydata_sphinx_theme/static
has compiled versions of these assets (e.g. CSS files). This folder is not tracked in.git
history, but it is bundled with the theme’s distribution.
Vendored scripts#
We vendor several packages in addition to our own CSS and JS.
For example, Bootstrap, JQuery, and Popper.
This is configured in the webpack.config.js
file, and imported in the respective SCSS
or JS
file in our assets folder.
FontAwesome icons#
Three “styles” of the FontAwesome 5 Free
icon font are used for icon links and admonitions, and is
the only vendored
font.
It is managed as a dependency in
package.json
Copied directly into the site statics at compilation, including licenses
Partially preloaded to reduce flicker and artifacts of early icon renders
Configured in
webpack.config.js
Jinja macros#
Our Webpack build generates a collection of Jinja macros in the static/webpack-macros.html
file.
These macros are imported in the main layout.html
file, and then inserted at various places in the page to link the static assets.
Some of the assets are “preloaded”, meaning that the browser begins requesting these resources before they’re actually needed.
In particular, our JavaScript assets are preloaded in <head>
, and the scripts are actually loaded at the end of <body>
.
Accessibility checks#
The accessibility checking tools can find a number of common HTML patterns which assistive technology can’t help users understand.
In addition to Lighthouse
in CI, the pa11y
stack is installed as part of the development environment.
The key components are:
pa11y which uses a headless browser to analyze an HTML page with a configurable set of rules based on publish standards
Pa11y-CI runs
pa11y
on multiple pagespa11y-reporter-html generates some nice HTML reports, suitable for review
Note
Presently, the default pa11y
ruleset, WCAG2AA
is used, a subset of
the Web Content Accessibility Guidelines.
The Quick Reference may provide
lighter reading.
Errors in CI/CD and what to do#
We have a list of known accessibility problems in the file docs/scripts/a11y-roadmap.txt
.
This contains a list of errors that we aim to fix in the future, and that do not cause tests to fail.
When a pa11y accessibility audit is run in our CI/CD, it checks for any errors that are not on this list, and if it finds them it will cause the job to error.
When you see an error in your CI/CD job, look at the logs under the Run accessibility audit
job.
You should see an output that looks like this:
JSON: /tmp/pa11y/pa11y-864/pa11y-ci-results.json
Roadmap: /home/runner/work/pydata-sphinx-theme/pydata-sphinx-theme/docs/a11y-roadmap.txt
not on roadmap:
WCAG2AA.Principle2.Guideline2_4.2_4_1.G1,G123,G124.NoSuchID: 4
on roadmap:
WCAG2AA.Principle1.Guideline1_3.1_3_1.H39.3.LayoutTable: 1
WCAG2AA.Principle1.Guideline1_3.1_3_1.H43,H63: 1
WCAG2AA.Principle1.Guideline1_3.1_3_1.H43.HeadersRequired: 1
WCAG2AA.Principle1.Guideline1_4.1_4_3.G18.Fail: 1828
WCAG2AA.Principle3.Guideline3_2.3_2_2.H32.2: 48
WCAG2AA.Principle4.Guideline4_1.4_1_2.H91.A.EmptyNoId: 9
passed: false
total errors: 1892
The problems that caused an error are in the not on roadmap
section.
Anything that is “not on the roadmap” is an error we have unexpectedly introduced in the PR.
These should be identified and fixed.
Fix accessibility errors#
We keep a list of known accessibility issues in the accessibility roadmap
.
These are issues which are currently flagged by the toolset, but that have not yet
been fixed.
To start working on one of the accessibility roadmap items, comment out one of the
lines in docs/a11y-roadmap.txt
, and re-run the audit to establish a baseline.
Then, fix the issue in either the HTML templates, CSS, or python code, and re-run the audit until it is fixed.
Run an accessibility audit locally#
To run the accessibility problem finder locally:
$ nox -s compile # Compile the theme assets
$ nox -s docs # Build the documentation
$ python docs/scripts/a11y.py # Run a helper script for an accessibility audit
The output of the last command includes:
a short summary of the current state of the accessibility rules we are trying to maintain
local paths to JSON and HTML reports which contain all of the issues found
Update our kitchen sink documents#
The kitchen sink reference is for demonstrating as much syntax and style for Sphinx builds as possible.
It is copied directly from the sphinx-themes.org
documentation so that we use standardized reference docs compared with other communities.
The source files for these pages are stored in the sphinx-themes.org
repository.
If you’d like to update our local files with any changes that have been made to the sphinx-themes.org
files, simply copy/paste those changes into our local files and make a commit.
Here’s a list of our pages and where they come from in sphinx-themes.org
:
Note
To demonstrate extra styles and syntax that is not in the Kitchen sink, use the Theme Elements reference.
Set up a manual development environment#
If you prefer not to use automation tools like nox
, or want to have more control over the specific version of packages that you’d like like installed, you may also manually set up a development environment locally.
To do so, follow the instructions on this page.
Create a new development environment#
This is optional, but it’s best to start with a fresh development environment so that you’ve isolated the packages that you’re using for this repository.
To do so, use a tool like conda, mamba, or virtualenv.
Clone the repository locally#
First clone this repository from the pydata
organization, or from a fork that you have created:
$ git clone https://github.com/pydata/pydata-sphinx-theme
$ cd pydata-sphinx-theme
Install the sphinx-theme-builder
#
We use the sphinx-theme-builder
to install nodejs
locally and to compile all CSS and JS assets needed for the theme.
Install it like so (note the cli
option so that we can run it from the command line):
$ pip install sphinx-theme-builder[cli]
Install this theme locally#
Next, install this theme locally so that we have the necessary dependencies to build the documentation and testing suite:
$ pip install -e .[dev]
Note that the sphinx-theme-builder
will automatically install a local copy of nodejs
for building the theme’s assets.
This will be placed in a .nodeenv
folder.
Build the documentation#
To manually build the documentation, run the following command:
$ sphinx-build docs docs/_build/html
Compile web assets (JS/CSS)#
To compile the javascript and CSS assets for the theme, run the following command:
$ stb compile
This will compile everything in the src/pydata_sphinx_theme/assets
folder and place them in the appropriate places in our theme’s folder structure.
Start a live-server to build and serve your documentation#
To manually open a server to watch your documentation for changes, build them, and display them locally in a browser, run this command:
$ stb serve docs --open-browser
Run the tests#
To manually run the tests for this theme, first set up your environment locally, and then run:
$ pytest
Demo site#
This is a simple demonstration site to show off a few visual and structural elements of the theme. Click the sections on the left sidebar to see how various elements look on this theme.
Kitchen Sink#
This section showcases the various elements that Sphinx supports out-of-the-box.
Paragraph Level Markup#
Inline Markup#
Paragraphs contain text and may contain inline markup: emphasis, strong emphasis, inline literals
,
standalone hyperlinks (http://www.python.org), external hyperlinks (Python 5), internal cross-references (example),
external hyperlinks with embedded URIs (Python web site), footnote references
(manually numbered 1, anonymous auto-numbered 3, labeled auto-numbered 2, or symbolic *),
citation references (12), substitution references (), and inline hyperlink targets
(see Targets below for a reference back to here). Character-level inline markup is also possible
(although exceedingly ugly!) in re
Structured
Text. Problems are indicated by |problematic|
text (generated by processing errors; this one is intentional).
Also with sphinx.ext.autodoc
, which I use in the demo, I can link to test_py_module.test.Foo
.
It will link you right to my code documentation for it.
The default role for interpreted text is Title Reference. Here are some explicit interpreted text roles:
a PEP reference (PEP 287); an RFC reference (RFC 2822); a subscript; a superscript;
and explicit roles for standard inline markup
.
GUI labels are a useful way to indicate that Some action is to be taken by the user.
The GUI label should not run over line-height
so as not to interfere with text from adjacent lines.
Key-bindings indicate that the read is to press a button on the keyboard or mouse,
for example MMB and Shift-MMB. Another useful markup to indicate a user action
is to use menuselection
this can be used to show short and long menus in software.
For example, and menuselection
can be seen here that breaks is too long to fit on this line.
.
Let’s test wrapping and whitespace significance in inline literals:
This is an example of --inline-literal --text, --including some--
strangely--hyphenated-words. Adjust-the-width-of-your-browser-window
to see how the text is wrapped. -- ---- -------- Now note the
spacing between the words of this sentence (words
should be grouped in pairs).
If the --pep-references
option was supplied, there should be a live link to PEP 258 here.
Math#
This is a test. Here is an equation: \(X_{0:5} = (X_0, X_1, X_2, X_3, X_4)\). Here is another:
You can add a link to equations like the one above (1) by using :eq:
.
Meta#
Blocks#
Literal Blocks#
Literal blocks are indicated with a double-colon (“::”) at the end of
the preceding paragraph (over there -->
). They can be indented:
if literal_block:
text = 'is left as-is'
spaces_and_linebreaks = 'are preserved'
markup_processing = None
Or they can be quoted without indentation:
>> Great idea!
>
> Why didn't I think of that?
Line Blocks#
Take it away, Eric the Orchestra Leader!
A one, two, a one two three fourHalf a bee, philosophically,must, ipso facto, half not be.But half the bee has got to be,vis a vis its entity. D’you see?But can a bee be said to beor not to be an entire bee,when half the bee is not a bee,due to some ancient injury?Singing…
Block Quotes#
Block quotes consist of indented body elements:
My theory by A. Elk. Brackets Miss, brackets. This theory goes as follows and begins now. All brontosauruses are thin at one end, much much thicker in the middle and then thin again at the far end. That is my theory, it is mine, and belongs to me and I own it, and what it is too.
—Anne Elk (Miss)
Doctest Blocks#
>>> print 'Python-specific usage examples; begun with ">>>"'
Python-specific usage examples; begun with ">>>"
>>> print '(cut and pasted from interactive Python sessions)'
(cut and pasted from interactive Python sessions)
Code Blocks#
# parsed-literal test curl -O http://someurl/release-.tar-gz
{
"windows": [
{
"panes": [
{
"shell_command": [
"echo 'did you know'",
"echo 'you can inline'"
]
},
{
"shell_command": "echo 'single commands'"
},
"echo 'for panes'"
],
"window_name": "long form"
}
],
"session_name": "shorthands"
}
1def some_function():
2 interesting = False
3 print 'This line is highlighted.'
4 print 'This one is not...'
5 print '...but this one is.'
References#
Footnotes#
- 1(1,2)
A footnote contains body elements, consistently indented by at least 3 spaces.
This is the footnote’s second paragraph.
- 2(1,2)
Footnotes may be numbered, either manually (as in 1) or automatically using a “#”-prefixed label. This footnote has a label so it can be referred to from multiple places, both as a footnote reference (2) and as a hyperlink reference (label).
- 3
This footnote is numbered automatically and anonymously using a label of “#” only.
- *
Footnotes may also use symbols, specified with a “*” label. Here’s a reference to the next footnote: †.
- †
This footnote shows the next symbol in the sequence.
- 4
Here’s an unreferenced footnote, with a reference to a nonexistent footnote: [5]_.
Citations#
- 11
This is the citation I made, let’s make this extremely long so that we can tell that it doesn’t follow the normal responsive table stuff.
- 12(1,2)
This citation has some
code blocks
in it, maybe some bold and italics too. Heck, lets put a link to a meta citation 13 too.- 13
This citation will have two backlinks.
Here’s a reference to the above, 12, and a [nonexistent] citation.
Here is another type of citation: citation
Glossary#
This is a glossary with definition terms for thing like Writing:
Targets#
This paragraph is pointed to by the explicit “example” target. A reference can be found under Inline Markup, above. Inline hyperlink targets are also possible.
Section headers are implicit targets, referred to by name. See Targets, which is a subsection of `Body Elements`_.
Explicit external targets are interpolated into references such as “Python 5”.
Targets may be indirect and anonymous. Thus this phrase may also refer to the Targets section.
Here’s a `hyperlink reference without a target`_, which generates an error.
Directives#
Above mini table of contents is generated by the .. contents::
directive.
It and all the other directives presented here are just a sample of the many
reStructuredText Directives. For others, please see:
http://docutils.sourceforge.net/docs/ref/rst/directives.html.
Centered text#
You can create a statement with centered text with .. centered::
This is centered text!
Images & Figures#
An image directive (also clickable – a hyperlink reference):
A figure is an image with a caption and/or a legend:#
re |
Revised, revisited, based on ‘re’ module. |
Structured |
Structure-enhanced text, structuredtext. |
Text |
Well it is, isn’t it? |
This paragraph is also part of the legend.
A figure directive with center alignment
This caption should be centered.#
Admonitions#
Attention
Directives at large.
Caution
Don’t take any wooden nickels.
Danger
Mad scientist at work!
Error
Does not compute.
Hint
It’s bigger than a bread box.
Important
Wash behind your ears.
Clean up your room.
Including the closet.
The bathroom too.
Take the trash out of the bathroom.
Clean the sink.
Call your mother.
Back up your data.
Note
This is a note. Equations within a note: \(G_{\mu\nu} = 8 \pi G (T_{\mu\nu} + \rho_\Lambda g_{\mu\nu})\).
Tip
15% if the service is good.
Example |
---|
Thing1 |
Thing2 |
Thing3 |
Warning
Strong prose may provoke extreme mental exertion. Reader discretion is strongly advised.
And, by the way…
You can make up your own admonition too.
Deprecations#
New in version v0.1: This is a version added message.
Changed in version v0.2: This is a version changed message.
Deprecated since version v0.3: This is a deprecation message.
Target Footnotes#
Replacement Text#
I recommend you try Python, the best language around 5.
Compound Paragraph#
This paragraph contains a literal block:
Connecting... OK
Transmitting data... OK
Disconnecting... OK
and thus consists of a simple paragraph, a literal block, and another simple paragraph. Nonetheless it is semantically one paragraph.
This construct is called a compound paragraph and can be produced with the “compound” directive.
Download Links#
API documentation#
asyncio
#
The asyncio package, tracking PEP 3156.
- class asyncio.AbstractEventLoop#
Abstract event loop.
- close()#
Close the loop.
The loop should not be running.
This is idempotent and irreversible.
No other methods should be called after this one.
- async connect_read_pipe(protocol_factory, pipe)#
Register read pipe in event loop. Set the pipe to non-blocking mode.
protocol_factory should instantiate object with Protocol interface. pipe is a file-like object. Return pair (transport, protocol), where transport supports the ReadTransport interface.
- async connect_write_pipe(protocol_factory, pipe)#
Register write pipe in event loop.
protocol_factory should instantiate object with BaseProtocol interface. Pipe is file-like object already switched to nonblocking. Return pair (transport, protocol), where transport support WriteTransport interface.
- async create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, *, family=0, proto=0, flags=0, reuse_address=None, reuse_port=None, allow_broadcast=None, sock=None)#
A coroutine which creates a datagram endpoint.
This method will try to establish the endpoint in the background. When successful, the coroutine returns a (transport, protocol) pair.
protocol_factory must be a callable returning a protocol instance.
socket family AF_INET, socket.AF_INET6 or socket.AF_UNIX depending on host (or family if specified), socket type SOCK_DGRAM.
reuse_address tells the kernel to reuse a local socket in TIME_WAIT state, without waiting for its natural timeout to expire. If not specified it will automatically be set to True on UNIX.
reuse_port tells the kernel to allow this endpoint to be bound to the same port as other existing endpoints are bound to, so long as they all set this flag when being created. This option is not supported on Windows and some UNIX’s. If the
SO_REUSEPORT
constant is not defined then this capability is unsupported.allow_broadcast tells the kernel to allow this endpoint to send messages to the broadcast address.
sock can optionally be specified in order to use a preexisting socket object.
- async create_server(protocol_factory, host=None, port=None, *, family=AddressFamily.AF_UNSPEC, flags=AddressInfo.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None, reuse_port=None, ssl_handshake_timeout=None, start_serving=True)#
A coroutine which creates a TCP server bound to host and port.
The return value is a Server object which can be used to stop the service.
If host is an empty string or None all interfaces are assumed and a list of multiple sockets will be returned (most likely one for IPv4 and another one for IPv6). The host parameter can also be a sequence (e.g. list) of hosts to bind to.
family can be set to either AF_INET or AF_INET6 to force the socket to use IPv4 or IPv6. If not set it will be determined from host (defaults to AF_UNSPEC).
flags is a bitmask for getaddrinfo().
sock can optionally be specified in order to use a preexisting socket object.
backlog is the maximum number of queued connections passed to listen() (defaults to 100).
ssl can be set to an SSLContext to enable SSL over the accepted connections.
reuse_address tells the kernel to reuse a local socket in TIME_WAIT state, without waiting for its natural timeout to expire. If not specified will automatically be set to True on UNIX.
reuse_port tells the kernel to allow this endpoint to be bound to the same port as other existing endpoints are bound to, so long as they all set this flag when being created. This option is not supported on Windows.
ssl_handshake_timeout is the time in seconds that an SSL server will wait for completion of the SSL handshake before aborting the connection. Default is 60s.
start_serving set to True (default) causes the created server to start accepting connections immediately. When set to False, the user should await Server.start_serving() or Server.serve_forever() to make the server to start accepting connections.
- async create_unix_server(protocol_factory, path=None, *, sock=None, backlog=100, ssl=None, ssl_handshake_timeout=None, start_serving=True)#
A coroutine which creates a UNIX Domain Socket server.
The return value is a Server object, which can be used to stop the service.
path is a str, representing a file systsem path to bind the server socket to.
sock can optionally be specified in order to use a preexisting socket object.
backlog is the maximum number of queued connections passed to listen() (defaults to 100).
ssl can be set to an SSLContext to enable SSL over the accepted connections.
ssl_handshake_timeout is the time in seconds that an SSL server will wait for the SSL handshake to complete (defaults to 60s).
start_serving set to True (default) causes the created server to start accepting connections immediately. When set to False, the user should await Server.start_serving() or Server.serve_forever() to make the server to start accepting connections.
- is_closed()#
Returns True if the event loop was closed.
- is_running()#
Return whether the event loop is currently running.
- run_forever()#
Run the event loop until stop() is called.
- run_until_complete(future)#
Run the event loop until a Future is done.
Return the Future’s result, or raise its exception.
- async sendfile(transport, file, offset=0, count=None, *, fallback=True)#
Send a file through a transport.
Return an amount of sent bytes.
- async shutdown_asyncgens()#
Shutdown all active asynchronous generators.
- async start_tls(transport, protocol, sslcontext, *, server_side=False, server_hostname=None, ssl_handshake_timeout=None)#
Upgrade a transport to TLS.
Return a new transport that protocol should start using immediately.
- stop()#
Stop the event loop as soon as reasonable.
Exactly how soon that is may depend on the implementation, but no more I/O callbacks should be scheduled.
- asyncio.gather(*coros_or_futures, loop=None, return_exceptions=False)#
Return a future aggregating results from the given coroutines/futures.
Coroutines will be wrapped in a future and scheduled in the event loop. They will not necessarily be scheduled in the same order as passed in.
All futures must share the same event loop. If all the tasks are done successfully, the returned future’s result is the list of results (in the order of the original sequence, not necessarily the order of results arrival). If return_exceptions is True, exceptions in the tasks are treated the same as successful results, and gathered in the result list; otherwise, the first raised exception will be immediately propagated to the returned future.
Cancellation: if the outer Future is cancelled, all children (that have not completed yet) are also cancelled. If any child is cancelled, this is treated as if it raised CancelledError – the outer Future is not cancelled in this case. (This is to prevent the cancellation of one child to cause other children to be cancelled.)
- asyncio.run(main, *, debug=False)#
Execute the coroutine and return the result.
This function runs the passed coroutine, taking care of managing the asyncio event loop and finalizing asynchronous generators.
This function cannot be called when another asyncio event loop is running in the same thread.
If debug is True, the event loop will be run in debug mode.
This function always creates a new event loop and closes it at the end. It should be used as a main entry point for asyncio programs, and should ideally only be called once.
Example:
- async def main():
await asyncio.sleep(1) print(‘hello’)
asyncio.run(main())
Lists & Tables#
Lists#
Enumerated Lists#
Arabic numerals.
lower alpha)
(lower roman)
upper alpha.
upper roman)
Lists that don’t start at 1:
Three
Four
C
D
iii
iv
List items may also be auto-enumerated.
Definition Lists#
- Term
Definition
- Termclassifier
Definition paragraph 1.
Definition paragraph 2.
- Term
Definition
I have no clue why the definition list below is classified as a different style of definition list than the one above.
- Is it the spaces in the term?
Maybe it was the multiple line paragraph in the line below that caused this?
- Is it the paragraph above the list maybe?
I guess a lot of these lists don’t have leading paragraphs?
- Is it everything all at once?
Who knows?!
Option Lists#
For listing command-line options:
- -a
command-line option “a”
- -b file
options can have arguments and long descriptions
- --long
options can be long also
- --input=file
long options can also have arguments
- --very-long-option
The description can also start on the next line.
The description may contain multiple body elements, regardless of where it starts.
- -x, -y, -z
Multiple options are an “option group”.
- -v, --verbose
Commonly-seen: short & long options.
- -1 file, --one=file, --two file
Multiple options with arguments.
- /V
DOS/VMS-style options too
There must be at least two spaces between the option and the description.
Field list#
- Author
David Goodger
- Address
123 Example Street Example, EX Canada A1B 2C3
- Contact
- Authors
Me; Myself; I
- organization
humankind
- date
$Date: 2012-01-03 19:23:53 +0000 (Tue, 03 Jan 2012) $
- status
This is a “work in progress”
- revision
$Revision: 7302 $
- version
1
- copyright
This document has been placed in the public domain. You may do with it as you wish. You may copy, modify, redistribute, reattribute, sell, buy, rent, lease, destroy, or improve it, quote it at length, excerpt, incorporate, collate, fold, staple, or mutilate it, or do anything else to it that your or anyone else’s heart desires.
- field name
This is a generic bibliographic field.
- field name 2
Generic bibliographic fields may contain multiple body elements.
Like this.
- Dedication
For Docutils users & co-developers.
- abstract
This document is a demonstration of the reStructuredText markup language, containing examples of all basic reStructuredText constructs and many advanced constructs.
Bullet Lists#
A simple list.
There are no margins between list items.
Simple lists do not contain multiple paragraphs. That’s a complex list.
In the case of a nested list
There are no margins between elements
Still no margins
Still no margins
A bullet list
Nested bullet list.
Nested item 2.
Item 2.
Paragraph 2 of item 2.
Nested bullet list.
Nested item 2.
Third level.
Item 2.
Nested item 3.
inline literall
inline literall
inline literall
This item has multiple paragraphs.
This item has multiple paragraphs.
This item has multiple paragraphs.
This item has multiple paragraphs.
here is a list in a second-level section.
-
here is an inner bullet
oh
one more
with an inline literally
. yahooheh heh. child. try to beat this embed:
1.. DOWNLOADED FROM sphinx-themes.org, DO NOT MANUALLY EDIT 2***************** 3API documentation 4***************** 5 6``asyncio`` 7=========== 8 9.. automodule:: asyncio 10 :members: run, gather, AbstractEventLoop
and another. yahoo
hi
how about an admonition?
Note
This is a note nested in a list.
and hehe
Hlists#
|
|
Hlist with images
|
|
Numbered List#
One,
Two.
Three with long text. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed feugiat sagittis neque quis eleifend. Duis rutrum lectus sit amet mattis suscipit.
Using bullets and letters. (A)
Using bullets and letters. (B)
Using bullets and letters. (C)
Tables#
Grid Tables#
Here’s a grid table followed by a simple table:
Header row, column 1 (header rows optional) |
Header 2 |
Header 3 |
Header 4 |
---|---|---|---|
body row 1, column 1 |
column 2 |
column 3 |
column 4 |
body row 2 |
Cells may span columns. |
||
body row 3 |
Cells may span rows. |
|
|
body row 4 |
|||
body row 5 |
Cells may also be
empty: |
Inputs |
Output |
|
---|---|---|
A |
B |
A or B |
False |
False |
False |
True |
False |
True |
False |
True |
True |
True |
True |
True |
Header 1 |
Header 2 |
Header 3 |
Header 1 |
Header 2 |
Header 3 |
Header 1 |
Header 2 |
Header 3 |
Header 1 |
Header 2 |
Header 3 |
---|---|---|---|---|---|---|---|---|---|---|---|
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
body row 1 |
column 2 |
column 3 |
List Tables#
List table |
Header 1 |
Header 2 |
Header 3 long. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam sit amet mauris arcu. |
---|---|---|---|
Stub Row 1 |
Row 1 |
Column 2 |
Column 3 long. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam sit amet mauris arcu. |
Stub Row 2 |
Row 2 |
Column 2 |
Column 3 long. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam sit amet mauris arcu. |
Stub Row 3 |
Row 3 |
Column 2 |
Column 3 long. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam sit amet mauris arcu. |
This is a short caption for a figure.# |
This is a long caption for a figure. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec porttitor dolor in odio posuere, vitae ornare libero mattis. In lobortis justo vestibulum nibh aliquet, non.# |
Theme-specific elements#
There are a few elements that are unique or particularly important to this theme. This page is a reference for how these look.
Embedding in admonitions#
Note
Here’s a note with:
A nested list
List item two
As well as:
Warning
A nested warning block to test nested admonitions.
Version changes#
You can write in your documentation when something has been changed, added or deprecated from one version to another.
New in version 0.1.1: Something is new, use it from now.
Changed in version 0.1.1: Something is modified, check your version number.
Deprecated since version 0.1.1: Something is deprecated, use something else instead.
HTML elements#
There are some libraries in the PyData ecosystem that use HTML and require their own styling. This section shows a few examples.
Plotly#
The HTML below shouldn’t display, but it uses RequireJS to make sure that all works as expected. If the widgets don’t show up, RequireJS may be broken.
import plotly.io as pio
import plotly.express as px
import plotly.offline as py
pio.renderers.default = "notebook"
df = px.data.iris()
fig = px.scatter(df, x="sepal_width", y="sepal_length", color="species", size="sepal_length")
fig
Xarray#
Here we demonstrate xarray
to ensure that it shows up properly.
import xarray as xr
import numpy as np
data = xr.DataArray(
np.random.randn(2, 3),
dims=("x", "y"),
coords={"x": [10, 20]}, attrs={"foo": "bar"}
)
data
<xarray.DataArray (x: 2, y: 3)> array([[-0.07247363, -0.09447655, 0.11564038], [-0.89803356, 1.22483231, -1.48215009]]) Coordinates: * x (x) int64 10 20 Dimensions without coordinates: y Attributes: foo: bar
Advanced API documentation and generated content#
This page contains general code elements that are common for package documentation.
Autosummary table and API stub pages#
|
Drop specified labels from rows or columns. |
|
Group DataFrame using a mapper or by a Series of columns. |
The ExtensionArray of the data backing this Series or Index. |
Inline module documentation#
numpy.linalg
#
numpy.linalg
#
The NumPy linear algebra functions rely on BLAS and LAPACK to provide efficient low level implementations of standard linear algebra algorithms. Those libraries may be provided by NumPy itself using C versions of a subset of their reference implementations but, when possible, highly optimized libraries that take advantage of specialized processor functionality are preferred. Examples of such libraries are OpenBLAS, MKL (TM), and ATLAS. Because those libraries are multithreaded and processor dependent, environmental variables and external packages such as threadpoolctl may be needed to control the number of threads or specify the processor architecture.
OpenBLAS: https://www.openblas.net/
threadpoolctl: https://github.com/joblib/threadpoolctl
Please note that the most-used linear algebra functions in NumPy are present in
the main numpy
namespace rather than in numpy.linalg
. There are:
dot
, vdot
, inner
, outer
, matmul
, tensordot
, einsum
,
einsum_path
and kron
.
Functions present in numpy.linalg are listed below.
multi_dot matrix_power
cholesky qr svd
eig eigh eigvals eigvalsh
norm cond det matrix_rank slogdet
solve tensorsolve lstsq inv pinv tensorinv
LinAlgError
- numpy.linalg.eig(a)#
Compute the eigenvalues and right eigenvectors of a square array.
- Parameters
- a(…, M, M) array
Matrices for which the eigenvalues and right eigenvectors will be computed
- Returns
- w(…, M) array
The eigenvalues, each repeated according to its multiplicity. The eigenvalues are not necessarily ordered. The resulting array will be of complex type, unless the imaginary part is zero in which case it will be cast to a real type. When a is real the resulting eigenvalues will be real (0 imaginary part) or occur in conjugate pairs
- v(…, M, M) array
The normalized (unit “length”) eigenvectors, such that the column
v[:,i]
is the eigenvector corresponding to the eigenvaluew[i]
.
- Raises
- LinAlgError
If the eigenvalue computation does not converge.
See also
eigvals
eigenvalues of a non-symmetric array.
eigh
eigenvalues and eigenvectors of a real symmetric or complex Hermitian (conjugate symmetric) array.
eigvalsh
eigenvalues of a real symmetric or complex Hermitian (conjugate symmetric) array.
scipy.linalg.eig
Similar function in SciPy that also solves the generalized eigenvalue problem.
scipy.linalg.schur
Best choice for unitary and other non-Hermitian normal matrices.
Notes
New in version 1.8.0.
Broadcasting rules apply, see the numpy.linalg documentation for details.
This is implemented using the
_geev
LAPACK routines which compute the eigenvalues and eigenvectors of general square arrays.The number w is an eigenvalue of a if there exists a vector v such that
a @ v = w * v
. Thus, the arrays a, w, and v satisfy the equationsa @ v[:,i] = w[i] * v[:,i]
for \(i \in \{0,...,M-1\}\).The array v of eigenvectors may not be of maximum rank, that is, some of the columns may be linearly dependent, although round-off error may obscure that fact. If the eigenvalues are all different, then theoretically the eigenvectors are linearly independent and a can be diagonalized by a similarity transformation using v, i.e,
inv(v) @ a @ v
is diagonal.For non-Hermitian normal matrices the SciPy function scipy.linalg.schur is preferred because the matrix v is guaranteed to be unitary, which is not the case when using eig. The Schur factorization produces an upper triangular matrix rather than a diagonal matrix, but for normal matrices only the diagonal of the upper triangular matrix is needed, the rest is roundoff error.
Finally, it is emphasized that v consists of the right (as in right-hand side) eigenvectors of a. A vector y satisfying
y.T @ a = z * y.T
for some number z is called a left eigenvector of a, and, in general, the left and right eigenvectors of a matrix are not necessarily the (perhaps conjugate) transposes of each other.References
G. Strang, Linear Algebra and Its Applications, 2nd Ed., Orlando, FL, Academic Press, Inc., 1980, Various pp.
Examples
>>> from numpy import linalg as LA
(Almost) trivial example with real e-values and e-vectors.
>>> w, v = LA.eig(np.diag((1, 2, 3))) >>> w; v array([1., 2., 3.]) array([[1., 0., 0.], [0., 1., 0.], [0., 0., 1.]])
Real matrix possessing complex e-values and e-vectors; note that the e-values are complex conjugates of each other.
>>> w, v = LA.eig(np.array([[1, -1], [1, 1]])) >>> w; v array([1.+1.j, 1.-1.j]) array([[0.70710678+0.j , 0.70710678-0.j ], [0. -0.70710678j, 0. +0.70710678j]])
Complex-valued matrix with real e-values (but complex-valued e-vectors); note that
a.conj().T == a
, i.e., a is Hermitian.>>> a = np.array([[1, 1j], [-1j, 1]]) >>> w, v = LA.eig(a) >>> w; v array([2.+0.j, 0.+0.j]) array([[ 0. +0.70710678j, 0.70710678+0.j ], # may vary [ 0.70710678+0.j , -0. +0.70710678j]])
Be careful about round-off error!
>>> a = np.array([[1 + 1e-9, 0], [0, 1 - 1e-9]]) >>> # Theor. e-values are 1 +/- 1e-9 >>> w, v = LA.eig(a) >>> w; v array([1., 1.]) array([[1., 0.], [0., 1.]])
- numpy.linalg.matrix_power(a, n)#
Raise a square matrix to the (integer) power n.
For positive integers n, the power is computed by repeated matrix squarings and matrix multiplications. If
n == 0
, the identity matrix of the same shape as M is returned. Ifn < 0
, the inverse is computed and then raised to theabs(n)
.Note
Stacks of object matrices are not currently supported.
- Parameters
- a(…, M, M) array_like
Matrix to be “powered”.
- nint
The exponent can be any integer or long integer, positive, negative, or zero.
- Returns
- a**n(…, M, M) ndarray or matrix object
The return value is the same shape and type as M; if the exponent is positive or zero then the type of the elements is the same as those of M. If the exponent is negative the elements are floating-point.
- Raises
- LinAlgError
For matrices that are not square or that (for negative powers) cannot be inverted numerically.
Examples
>>> from numpy.linalg import matrix_power >>> i = np.array([[0, 1], [-1, 0]]) # matrix equiv. of the imaginary unit >>> matrix_power(i, 3) # should = -i array([[ 0, -1], [ 1, 0]]) >>> matrix_power(i, 0) array([[1, 0], [0, 1]]) >>> matrix_power(i, -3) # should = 1/(-i) = i, but w/ f.p. elements array([[ 0., 1.], [-1., 0.]])
Somewhat more sophisticated example
>>> q = np.zeros((4, 4)) >>> q[0:2, 0:2] = -i >>> q[2:4, 2:4] = i >>> q # one of the three quaternion units not equal to 1 array([[ 0., -1., 0., 0.], [ 1., 0., 0., 0.], [ 0., 0., 0., 1.], [ 0., 0., -1., 0.]]) >>> matrix_power(q, 2) # = -np.eye(4) array([[-1., 0., 0., 0.], [ 0., -1., 0., 0.], [ 0., 0., -1., 0.], [ 0., 0., 0., -1.]])
- numpy.linalg.norm(x, ord=None, axis=None, keepdims=False)#
Matrix or vector norm.
This function is able to return one of eight different matrix norms, or one of an infinite number of vector norms (described below), depending on the value of the
ord
parameter.- Parameters
- xarray_like
Input array. If axis is None, x must be 1-D or 2-D, unless ord is None. If both axis and ord are None, the 2-norm of
x.ravel
will be returned.- ord{non-zero int, inf, -inf, ‘fro’, ‘nuc’}, optional
Order of the norm (see table under
Notes
). inf means numpy’s inf object. The default is None.- axis{None, int, 2-tuple of ints}, optional.
If axis is an integer, it specifies the axis of x along which to compute the vector norms. If axis is a 2-tuple, it specifies the axes that hold 2-D matrices, and the matrix norms of these matrices are computed. If axis is None then either a vector norm (when x is 1-D) or a matrix norm (when x is 2-D) is returned. The default is None.
New in version 1.8.0.
- keepdimsbool, optional
If this is set to True, the axes which are normed over are left in the result as dimensions with size one. With this option the result will broadcast correctly against the original x.
New in version 1.10.0.
- Returns
- nfloat or ndarray
Norm of the matrix or vector(s).
See also
scipy.linalg.norm
Similar function in SciPy.
Notes
For values of
ord < 1
, the result is, strictly speaking, not a mathematical ‘norm’, but it may still be useful for various numerical purposes.The following norms can be calculated:
ord
norm for matrices
norm for vectors
None
Frobenius norm
2-norm
‘fro’
Frobenius norm
–
‘nuc’
nuclear norm
–
inf
max(sum(abs(x), axis=1))
max(abs(x))
-inf
min(sum(abs(x), axis=1))
min(abs(x))
0
–
sum(x != 0)
1
max(sum(abs(x), axis=0))
as below
-1
min(sum(abs(x), axis=0))
as below
2
2-norm (largest sing. value)
as below
-2
smallest singular value
as below
other
–
sum(abs(x)**ord)**(1./ord)
The Frobenius norm is given by [1]:
\(||A||_F = [\sum_{i,j} abs(a_{i,j})^2]^{1/2}\)
The nuclear norm is the sum of the singular values.
Both the Frobenius and nuclear norm orders are only defined for matrices and raise a ValueError when
x.ndim != 2
.References
- 1
G. H. Golub and C. F. Van Loan, Matrix Computations, Baltimore, MD, Johns Hopkins University Press, 1985, pg. 15
Examples
>>> from numpy import linalg as LA >>> a = np.arange(9) - 4 >>> a array([-4, -3, -2, ..., 2, 3, 4]) >>> b = a.reshape((3, 3)) >>> b array([[-4, -3, -2], [-1, 0, 1], [ 2, 3, 4]])
>>> LA.norm(a) 7.745966692414834 >>> LA.norm(b) 7.745966692414834 >>> LA.norm(b, 'fro') 7.745966692414834 >>> LA.norm(a, np.inf) 4.0 >>> LA.norm(b, np.inf) 9.0 >>> LA.norm(a, -np.inf) 0.0 >>> LA.norm(b, -np.inf) 2.0
>>> LA.norm(a, 1) 20.0 >>> LA.norm(b, 1) 7.0 >>> LA.norm(a, -1) -4.6566128774142013e-010 >>> LA.norm(b, -1) 6.0 >>> LA.norm(a, 2) 7.745966692414834 >>> LA.norm(b, 2) 7.3484692283495345
>>> LA.norm(a, -2) 0.0 >>> LA.norm(b, -2) 1.8570331885190563e-016 # may vary >>> LA.norm(a, 3) 5.8480354764257312 # may vary >>> LA.norm(a, -3) 0.0
Using the axis argument to compute vector norms:
>>> c = np.array([[ 1, 2, 3], ... [-1, 1, 4]]) >>> LA.norm(c, axis=0) array([ 1.41421356, 2.23606798, 5. ]) >>> LA.norm(c, axis=1) array([ 3.74165739, 4.24264069]) >>> LA.norm(c, ord=1, axis=1) array([ 6., 6.])
Using the axis argument to compute matrix norms:
>>> m = np.arange(8).reshape(2,2,2) >>> LA.norm(m, axis=(1,2)) array([ 3.74165739, 11.22497216]) >>> LA.norm(m[0, :, :]), LA.norm(m[1, :, :]) (3.7416573867739413, 11.224972160321824)
- numpy.linalg.tensorinv(a, ind=2)#
Compute the ‘inverse’ of an N-dimensional array.
The result is an inverse for a relative to the tensordot operation
tensordot(a, b, ind)
, i. e., up to floating-point accuracy,tensordot(tensorinv(a), a, ind)
is the “identity” tensor for the tensordot operation.- Parameters
- aarray_like
Tensor to ‘invert’. Its shape must be ‘square’, i. e.,
prod(a.shape[:ind]) == prod(a.shape[ind:])
.- indint, optional
Number of first indices that are involved in the inverse sum. Must be a positive integer, default is 2.
- Returns
- bndarray
a’s tensordot inverse, shape
a.shape[ind:] + a.shape[:ind]
.
- Raises
- LinAlgError
If a is singular or not ‘square’ (in the above sense).
See also
numpy.tensordot
,tensorsolve
Examples
>>> a = np.eye(4*6) >>> a.shape = (4, 6, 8, 3) >>> ainv = np.linalg.tensorinv(a, ind=2) >>> ainv.shape (8, 3, 4, 6) >>> b = np.random.randn(4, 6) >>> np.allclose(np.tensordot(ainv, b), np.linalg.tensorsolve(a, b)) True
>>> a = np.eye(4*6) >>> a.shape = (24, 8, 3) >>> ainv = np.linalg.tensorinv(a, ind=1) >>> ainv.shape (8, 3, 24) >>> b = np.random.randn(24) >>> np.allclose(np.tensordot(ainv, b, 1), np.linalg.tensorsolve(a, b)) True
C++ API#
-
type MyType#
Some type
-
template<typename T, std::size_t N>
class std::array# Some cpp class
-
float Sphinx::version#
The description of Sphinx::version.
-
int version#
The description of version.
-
typedef std::vector<int> List#
The description of List type.
JavaScript API#
Link to
ModTopLevel()
- class module_a.submodule.ModTopLevel()#
Link to
mod_child_1()
Link to
ModTopLevel.mod_child_1()
- ModTopLevel.mod_child_1()#
Link to
mod_child_2()
- ModTopLevel.mod_child_2()#
Link to
ModTopLevel()
- class module_b.submodule.ModNested()#
- ModNested.nested_child_1()#
Link to
nested_child_2()
- ModNested.nested_child_2()#
Link to
nested_child_1()
Generated Index#
Part of the sphinx build process in generate and index file: Index.
Optional parameter args#
At this point optional parameters cannot be generated from code. However, some projects will manually do it, like so:
This example comes from django-payments module docs.
- class payments.dotpay.DotpayProvider(seller_id, pin[, channel=0[, lock=False], lang='pl'])#
This backend implements payments using a popular Polish gateway, Dotpay.pl.
Due to API limitations there is no support for transferring purchased items.
- Parameters
seller_id – Seller ID assigned by Dotpay
pin – PIN assigned by Dotpay
channel – Default payment channel (consult reference guide)
lang – UI language
lock – Whether to disable channels other than the default selected above
Data#
- numpy.linalg.Data_item_1#
- numpy.linalg.Data_item_2#
- numpy.linalg.Data_item_3#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce congue elit eu hendrerit mattis.
Some data link Data_item_1
.
Pandas example - Indexing and selecting data#
Note
This is an example page with excerpts from the pandas docs, for some “real world” content. But including it here apart from the rest of the pandas docs will mean that some of the links won’t work, and not all code examples are shown with their complete outputs.
The axis labeling information in pandas objects serves many purposes:
Identifies data (i.e. provides metadata) using known indicators, important for analysis, visualization, and interactive console display.
Enables automatic and explicit data alignment.
Allows intuitive getting and setting of subsets of the data set.
In this section, we will focus on the final point: namely, how to slice, dice, and generally get and set subsets of pandas objects. The primary focus will be on Series and DataFrame as they have received more development attention in this area.
Note
The Python and NumPy indexing operators []
and attribute operator .
provide quick and easy access to pandas data structures across a wide range
of use cases. This makes interactive work intuitive, as there’s little new
to learn if you already know how to deal with Python dictionaries and NumPy
arrays. However, since the type of the data to be accessed isn’t known in
advance, directly using standard operators has some optimization limits. For
production code, we recommended that you take advantage of the optimized
pandas data access methods exposed in this chapter.
Warning
Whether a copy or a reference is returned for a setting operation, may
depend on the context. This is sometimes called chained assignment
and
should be avoided. See Returning a View versus Copy.
Different choices for indexing#
Object selection has had a number of user-requested additions in order to support more explicit location based indexing. Pandas now supports three types of multi-axis indexing.
.loc
is primarily label based, but may also be used with a boolean array..loc
will raiseKeyError
when the items are not found. Allowed inputs are:A single label, e.g.
5
or'a'
(Note that5
is interpreted as a label of the index. This use is not an integer position along the index.).A list or array of labels
['a', 'b', 'c']
.A slice object with labels
'a':'f'
(Note that contrary to usual python slices, both the start and the stop are included, when present in the index!)A boolean array
A
callable
function with one argument (the calling Series or DataFrame) and that returns valid output for indexing (one of the above).
See more at Selection by Label.
.iloc
is primarily integer position based (from0
tolength-1
of the axis), but may also be used with a boolean array..iloc
will raiseIndexError
if a requested indexer is out-of-bounds, except slice indexers which allow out-of-bounds indexing. (this conforms with Python/NumPy slice semantics). Allowed inputs are:An integer e.g.
5
.A list or array of integers
[4, 3, 0]
.A slice object with ints
1:7
.A boolean array.
A
callable
function with one argument (the calling Series or DataFrame) and that returns valid output for indexing (one of the above).
.loc
,.iloc
, and also[]
indexing can accept acallable
as indexer. See more at Selection By Callable.
Getting values from an object with multi-axes selection uses the following
notation (using .loc
as an example, but the following applies to .iloc
as
well). Any of the axes accessors may be the null slice :
. Axes left out of
the specification are assumed to be :
, e.g. p.loc['a']
is equivalent to
p.loc['a', :, :]
.
Object Type |
Indexers |
---|---|
Series |
|
DataFrame |
|
Basics#
As mentioned when introducing the data structures in the last section,
the primary function of indexing with []
(a.k.a. __getitem__
for those familiar with implementing class behavior in Python) is selecting out
lower-dimensional slices. The following table shows return type values when
indexing pandas objects with []
:
Object Type |
Selection |
Return Value Type |
---|---|---|
Series |
|
scalar value |
DataFrame |
|
|
Here we construct a simple time series data set to use for illustrating the indexing functionality:
>>> dates = pd.date_range('1/1/2000', periods=8)
>>> df = pd.DataFrame(np.random.randn(8, 4),
... index=dates, columns=['A', 'B', 'C', 'D'])
...
>>> df
A B C D
2000-01-01 0.469112 -0.282863 -1.509059 -1.135632
2000-01-02 1.212112 -0.173215 0.119209 -1.044236
2000-01-03 -0.861849 -2.104569 -0.494929 1.071804
2000-01-04 0.721555 -0.706771 -1.039575 0.271860
2000-01-05 -0.424972 0.567020 0.276232 -1.087401
2000-01-06 -0.673690 0.113648 -1.478427 0.524988
2000-01-07 0.404705 0.577046 -1.715002 -1.039268
2000-01-08 -0.370647 -1.157892 -1.344312 0.844885
Note
None of the indexing functionality is time series specific unless specifically stated.
Thus, as per above, we have the most basic indexing using []
:
>>> s = df['A']
>>> s[dates[5]]
-0.6736897080883706
You can pass a list of columns to []
to select columns in that order.
If a column is not contained in the DataFrame, an exception will be
raised. Multiple columns can also be set in this manner:
>>> df
A B C D
2000-01-01 0.469112 -0.282863 -1.509059 -1.135632
2000-01-02 1.212112 -0.173215 0.119209 -1.044236
2000-01-03 -0.861849 -2.104569 -0.494929 1.071804
2000-01-04 0.721555 -0.706771 -1.039575 0.271860
2000-01-05 -0.424972 0.567020 0.276232 -1.087401
2000-01-06 -0.673690 0.113648 -1.478427 0.524988
2000-01-07 0.404705 0.577046 -1.715002 -1.039268
2000-01-08 -0.370647 -1.157892 -1.344312 0.844885
>>> df[['B', 'A']] = df[['A', 'B']]
>>> df
A B C D
2000-01-01 -0.282863 0.469112 -1.509059 -1.135632
2000-01-02 -0.173215 1.212112 0.119209 -1.044236
2000-01-03 -2.104569 -0.861849 -0.494929 1.071804
2000-01-04 -0.706771 0.721555 -1.039575 0.271860
2000-01-05 0.567020 -0.424972 0.276232 -1.087401
2000-01-06 0.113648 -0.673690 -1.478427 0.524988
2000-01-07 0.577046 0.404705 -1.715002 -1.039268
2000-01-08 -1.157892 -0.370647 -1.344312 0.844885
You may find this useful for applying a transform (in-place) to a subset of the columns.
Warning
pandas aligns all AXES when setting Series
and DataFrame
from .loc
, and .iloc
.
This will not modify df
because the column alignment is before value assignment.
>>> df[['A', 'B']]
A B
2000-01-01 -0.282863 0.469112
2000-01-02 -0.173215 1.212112
2000-01-03 -2.104569 -0.861849
2000-01-04 -0.706771 0.721555
2000-01-05 0.567020 -0.424972
2000-01-06 0.113648 -0.673690
2000-01-07 0.577046 0.404705
2000-01-08 -1.157892 -0.370647
>>> df.loc[:, ['B', 'A']] = df[['A', 'B']]
>>> df[['A', 'B']]
A B
2000-01-01 -0.282863 0.469112
2000-01-02 -0.173215 1.212112
2000-01-03 -2.104569 -0.861849
2000-01-04 -0.706771 0.721555
2000-01-05 0.567020 -0.424972
2000-01-06 0.113648 -0.673690
2000-01-07 0.577046 0.404705
2000-01-08 -1.157892 -0.370647
The correct way to swap column values is by using raw values:
>>> df.loc[:, ['B', 'A']] = df[['A', 'B']].to_numpy()
>>> df[['A', 'B']]
A B
2000-01-01 0.469112 -0.282863
2000-01-02 1.212112 -0.173215
2000-01-03 -0.861849 -2.104569
2000-01-04 0.721555 -0.706771
2000-01-05 -0.424972 0.567020
2000-01-06 -0.673690 0.113648
2000-01-07 0.404705 0.577046
2000-01-08 -0.370647 -1.157892
Attribute access#
You may access an index on a Series
or column on a DataFrame
directly
as an attribute:
sa = pd.Series([1, 2, 3], index=list('abc'))
dfa = df.copy()
sa.b
dfa.A
>>> sa.a = 5
>>> sa
a 5
b 2
c 3
dtype: int64
>>> dfa.A = list(range(len(dfa.index))) # ok if A already exists
>>> dfa
A B C D
2000-01-01 0 -0.282863 -1.509059 -1.135632
2000-01-02 1 -0.173215 0.119209 -1.044236
2000-01-03 2 -2.104569 -0.494929 1.071804
2000-01-04 3 -0.706771 -1.039575 0.271860
2000-01-05 4 0.567020 0.276232 -1.087401
2000-01-06 5 0.113648 -1.478427 0.524988
2000-01-07 6 0.577046 -1.715002 -1.039268
2000-01-08 7 -1.157892 -1.344312 0.844885
>>> dfa['A'] = list(range(len(dfa.index))) # use this form to create a new column
>>> dfa
A B C D
2000-01-01 0 -0.282863 -1.509059 -1.135632
2000-01-02 1 -0.173215 0.119209 -1.044236
2000-01-03 2 -2.104569 -0.494929 1.071804
2000-01-04 3 -0.706771 -1.039575 0.271860
2000-01-05 4 0.567020 0.276232 -1.087401
2000-01-06 5 0.113648 -1.478427 0.524988
2000-01-07 6 0.577046 -1.715002 -1.039268
2000-01-08 7 -1.157892 -1.344312 0.844885
Warning
You can use this access only if the index element is a valid Python identifier, e.g.
s.1
is not allowed. See here for an explanation of valid identifiers.The attribute will not be available if it conflicts with an existing method name, e.g.
s.min
is not allowed, buts['min']
is possible.Similarly, the attribute will not be available if it conflicts with any of the following list:
index
,major_axis
,minor_axis
,items
.In any of these cases, standard indexing will still work, e.g.
s['1']
,s['min']
, ands['index']
will access the corresponding element or column.
If you are using the IPython environment, you may also use tab-completion to see these accessible attributes.
You can also assign a dict
to a row of a DataFrame
:
>>> x = pd.DataFrame({'x': [1, 2, 3], 'y': [3, 4, 5]})
>>> x.iloc[1] = {'x': 9, 'y': 99}
>>> x
x y
0 1 3
1 9 99
2 3 5
You can use attribute access to modify an existing element of a Series or column of a DataFrame, but be careful;
if you try to use attribute access to create a new column, it creates a new attribute rather than a
new column. In 0.21.0 and later, this will raise a UserWarning
:
>>> df = pd.DataFrame({'one': [1., 2., 3.]})
>>> df.two = [4, 5, 6]
UserWarning: Pandas doesn't allow Series to be assigned into nonexistent columns - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute_access
>>> df
one
0 1.0
1 2.0
2 3.0
Selection by label#
Warning
Whether a copy or a reference is returned for a setting operation, may depend on the context.
This is sometimes called chained assignment
and should be avoided.
See Returning a View versus Copy.
Warning
.loc
is strict when you present slicers that are not compatible (or convertible) with the index type. For example using integers in aDatetimeIndex
. These will raise aTypeError
.
dfl = pd.DataFrame(np.random.randn(5, 4),
columns=list('ABCD'),
index=pd.date_range('20130101', periods=5))
dfl
>>> dfl.loc[2:3]
TypeError: cannot do slice indexing on <class 'pandas.tseries.index.DatetimeIndex'> with these indexers [2] of <type 'int'>
String likes in slicing can be convertible to the type of the index and lead to natural slicing.
dfl.loc['20130102':'20130104']
pandas provides a suite of methods in order to have purely label based indexing. This is a strict inclusion based protocol.
Every label asked for must be in the index, or a KeyError
will be raised.
When slicing, both the start bound AND the stop bound are included, if present in the index.
Integers are valid labels, but they refer to the label and not the position.
The .loc
attribute is the primary access method. The following are valid inputs:
A single label, e.g.
5
or'a'
(Note that5
is interpreted as a label of the index. This use is not an integer position along the index.).A list or array of labels
['a', 'b', 'c']
.A slice object with labels
'a':'f'
(Note that contrary to usual python slices, both the start and the stop are included, when present in the index! See Slicing with labels.A boolean array.
A
callable
, see Selection By Callable.
>>> s1 = pd.Series(np.random.randn(6), index=list('abcdef'))
>>> s1
a 1.431256
b 1.340309
c -1.170299
d -0.226169
e 0.410835
f 0.813850
dtype: float64
>>> s1.loc['c':]
c -1.170299
d -0.226169
e 0.410835
f 0.813850
dtype: float64
>>> s1.loc['b']
1.3403088497993827
Note that setting works as well:
>>> s1.loc['c':] = 0
>>> s1
a 1.431256
b 1.340309
c 0.000000
d 0.000000
e 0.000000
f 0.000000
dtype: float64
With a DataFrame:
>>> df1 = pd.DataFrame(np.random.randn(6, 4),
.... index=list('abcdef'),
.... columns=list('ABCD'))
....
>>> df1
A B C D
a 0.132003 -0.827317 -0.076467 -1.187678
b 1.130127 -1.436737 -1.413681 1.607920
c 1.024180 0.569605 0.875906 -2.211372
d 0.974466 -2.006747 -0.410001 -0.078638
e 0.545952 -1.219217 -1.226825 0.769804
f -1.281247 -0.727707 -0.121306 -0.097883
>>> df1.loc[['a', 'b', 'd'], :]
A B C D
a 0.132003 -0.827317 -0.076467 -1.187678
b 1.130127 -1.436737 -1.413681 1.607920
Slicing with labels#
When using .loc
with slices, if both the start and the stop labels are
present in the index, then elements located between the two (including them)
are returned:
>>> s = pd.Series(list('abcde'), index=[0, 3, 2, 5, 4])
>>> s.loc[3:5]
3 b
2 c
5 d
dtype: object
If at least one of the two is absent, but the index is sorted, and can be compared against start and stop labels, then slicing will still work as expected, by selecting labels which rank between the two:
>>> s.sort_index()
0 a
2 c
3 b
4 e
5 d
dtype: object
>>> s.sort_index().loc[1:6]
2 c
3 b
4 e
5 d
dtype: object
However, if at least one of the two is absent and the index is not sorted, an
error will be raised (since doing otherwise would be computationally expensive,
as well as potentially ambiguous for mixed type indexes). For instance, in the
above example, s.loc[1:6]
would raise KeyError
.
Selection by position#
Warning
Whether a copy or a reference is returned for a setting operation, may depend on the context.
This is sometimes called chained assignment
and should be avoided.
See Returning a View versus Copy.
Pandas provides a suite of methods in order to get purely integer based indexing. The semantics follow closely Python and NumPy slicing. These are 0-based
indexing. When slicing, the start bound is included, while the upper bound is excluded. Trying to use a non-integer, even a valid label will raise an IndexError
.
The .iloc
attribute is the primary access method. The following are valid inputs:
An integer e.g.
5
.A list or array of integers
[4, 3, 0]
.A slice object with ints
1:7
.A boolean array.
A
callable
, see Selection By Callable.
>>> s1 = pd.Series(np.random.randn(5), index=list(range(0, 10, 2)))
>>> s1
0 0.695775
2 0.341734
4 0.959726
6 -1.110336
8 -0.619976
dtype: float64
>>> s1.iloc[:3]
0 0.695775
2 0.341734
4 0.959726
dtype: float64
>>> s1.iloc[3]
-1.110336102891167
Note that setting works as well:
s1.iloc[:3] = 0
s1
With a DataFrame:
df1 = pd.DataFrame(np.random.randn(6, 4),
index=list(range(0, 12, 2)),
columns=list(range(0, 8, 2)))
df1
Select via integer slicing:
df1.iloc[:3]
df1.iloc[1:5, 2:4]
Select via integer list:
df1.iloc[[1, 3, 5], [1, 3]]
df1.iloc[1:3, :]
df1.iloc[:, 1:3]
# this is also equivalent to ``df1.iat[1,1]``
df1.iloc[1, 1]
For getting a cross section using an integer position (equiv to df.xs(1)
):
df1.iloc[1]
Out of range slice indexes are handled gracefully just as in Python/Numpy.
# these are allowed in python/numpy.
x = list('abcdef')
x
x[4:10]
x[8:10]
s = pd.Series(x)
s
s.iloc[4:10]
s.iloc[8:10]
Note that using slices that go out of bounds can result in an empty axis (e.g. an empty DataFrame being returned).
dfl = pd.DataFrame(np.random.randn(5, 2), columns=list('AB'))
dfl
dfl.iloc[:, 2:3]
dfl.iloc[:, 1:3]
dfl.iloc[4:6]
A single indexer that is out of bounds will raise an IndexError
.
A list of indexers where any element is out of bounds will raise an
IndexError
.
>>> dfl.iloc[[4, 5, 6]]
IndexError: positional indexers are out-of-bounds
>>> dfl.iloc[:, 4]
IndexError: single positional indexer is out-of-bounds
Selection by callable#
.loc
, .iloc
, and also []
indexing can accept a callable
as indexer.
The callable
must be a function with one argument (the calling Series or DataFrame) that returns valid output for indexing.
>>> df1 = pd.DataFrame(np.random.randn(6, 4),
.... index=list('abcdef'),
.... columns=list('ABCD'))
....
>>> df1
A B C D
a -0.023688 2.410179 1.450520 0.206053
b -0.251905 -2.213588 1.063327 1.266143
c 0.299368 -0.863838 0.408204 -1.048089
d -0.025747 -0.988387 0.094055 1.262731
e 1.289997 0.082423 -0.055758 0.536580
f -0.489682 0.369374 -0.034571 -2.484478
>>> df1.loc[lambda df: df['A'] > 0, :]
A B C D
c 0.299368 -0.863838 0.408204 -1.048089
e 1.289997 0.082423 -0.055758 0.536580
>>> df1.loc[:, lambda df: ['A', 'B']]
A B
a -0.023688 2.410179
b -0.251905 -2.213588
c 0.299368 -0.863838
d -0.025747 -0.988387
e 1.289997 0.082423
f -0.489682 0.369374
>>> df1.iloc[:, lambda df: [0, 1]]
A B
a -0.023688 2.410179
b -0.251905 -2.213588
c 0.299368 -0.863838
d -0.025747 -0.988387
e 1.289997 0.082423
f -0.489682 0.369374
>>> df1[lambda df: df.columns[0]]
a -0.023688
b -0.251905
c 0.299368
d -0.025747
e 1.289997
f -0.489682
Name: A, dtype: float64
You can use callable indexing in Series
.
df1['A'].loc[lambda s: s > 0]
Using these methods / indexers, you can chain data selection operations without using a temporary variable.
bb = pd.read_csv('data/baseball.csv', index_col='id')
(bb.groupby(['year', 'team']).sum()
.loc[lambda df: df['r'] > 100])
Boolean indexing#
Another common operation is the use of boolean vectors to filter the data.
The operators are: |
for or
, &
for and
, and ~
for not
.
These must be grouped by using parentheses, since by default Python will
evaluate an expression such as df['A'] > 2 & df['B'] < 3
as
df['A'] > (2 & df['B']) < 3
, while the desired evaluation order is
(df['A > 2) & (df['B'] < 3)
.
Using a boolean vector to index a Series works exactly as in a NumPy ndarray:
>>> s = pd.Series(range(-3, 4))
>>> s
0 -3
1 -2
2 -1
3 0
4 1
5 2
6 3
dtype: int64
>>> s[s > 0]
4 1
5 2
6 3
dtype: int64
>>> s[(s < -1) | (s > 0.5)]
0 -3
1 -2
4 1
5 2
6 3
dtype: int64
>>> s[~(s < 0)]
3 0
4 1
5 2
6 3
dtype: int64
You may select rows from a DataFrame using a boolean vector the same length as the DataFrame’s index (for example, something derived from one of the columns of the DataFrame):
df[df['A'] > 0]
List comprehensions and the map
method of Series can also be used to produce
more complex criteria:
df2 = pd.DataFrame({'a': ['one', 'one', 'two', 'three', 'two', 'one', 'six'],
'b': ['x', 'y', 'y', 'x', 'y', 'x', 'x'],
'c': np.random.randn(7)})
# only want 'two' or 'three'
criterion = df2['a'].map(lambda x: x.startswith('t'))
df2[criterion]
# equivalent but slower
df2[[x.startswith('t') for x in df2['a']]]
# Multiple criteria
df2[criterion & (df2['b'] == 'x')]
With the choice methods Selection by Label, Selection by Position you may select along more than one axis using boolean vectors combined with other indexing expressions.
df2.loc[criterion & (df2['b'] == 'x'), 'b':'c']
The query()
Method#
DataFrame
objects have a query()
method that allows selection using an expression.
You can get the value of the frame where column b
has values
between the values of columns a
and c
. For example:
n = 10
df = pd.DataFrame(np.random.rand(n, 3), columns=list('abc'))
df
# pure python
df[(df['a'] < df['b']) & (df['b'] < df['c'])]
# query
df.query('(a < b) & (b < c)')
Do the same thing but fall back on a named index if there is no column
with the name a
.
df = pd.DataFrame(np.random.randint(n / 2, size=(n, 2)), columns=list('bc'))
df.index.name = 'a'
df
df.query('a < b and b < c')
If instead you don’t want to or cannot name your index, you can use the name
index
in your query expression:
df = pd.DataFrame(np.random.randint(n, size=(n, 2)), columns=list('bc'))
df
df.query('index < b < c')
Note
If the name of your index overlaps with a column name, the column name is given precedence. For example,
df = pd.DataFrame({'a': np.random.randint(5, size=5)})
df.index.name = 'a'
df.query('a > 2') # uses the column 'a', not the index
You can still use the index in a query expression by using the special identifier ‘index’:
df.query('index > 2')
If for some reason you have a column named index
, then you can refer to
the index as ilevel_0
as well, but at this point you should consider
renaming your columns to something less ambiguous.
MultiIndex
query()
Syntax#
You can also use the levels of a DataFrame
with a
MultiIndex
as if they were columns in the frame:
n = 10
colors = np.random.choice(['red', 'green'], size=n)
foods = np.random.choice(['eggs', 'ham'], size=n)
colors
foods
index = pd.MultiIndex.from_arrays([colors, foods], names=['color', 'food'])
df = pd.DataFrame(np.random.randn(n, 2), index=index)
df
df.query('color == "red"')
If the levels of the MultiIndex
are unnamed, you can refer to them using
special names:
df.index.names = [None, None]
df
df.query('ilevel_0 == "red"')
The convention is ilevel_0
, which means “index level 0” for the 0th level
of the index
.
query()
Use Cases#
A use case for query()
is when you have a collection of
DataFrame
objects that have a subset of column names (or index
levels/names) in common. You can pass the same query to both frames without
having to specify which frame you’re interested in querying
df = pd.DataFrame(np.random.rand(n, 3), columns=list('abc'))
df
df2 = pd.DataFrame(np.random.rand(n + 2, 3), columns=df.columns)
df2
expr = '0.0 <= a <= c <= 0.5'
map(lambda frame: frame.query(expr), [df, df2])
query()
Python versus pandas Syntax Comparison#
Full numpy-like syntax:
df = pd.DataFrame(np.random.randint(n, size=(n, 3)), columns=list('abc'))
df
df.query('(a < b) & (b < c)')
df[(df['a'] < df['b']) & (df['b'] < df['c'])]
Slightly nicer by removing the parentheses (by binding making comparison
operators bind tighter than &
and |
).
df.query('a < b & b < c')
Use English instead of symbols:
df.query('a < b and b < c')
Pretty close to how you might write it on paper:
df.query('a < b < c')
The in
and not in
operators#
query()
also supports special use of Python’s in
and
not in
comparison operators, providing a succinct syntax for calling the
isin
method of a Series
or DataFrame
.
# get all rows where columns "a" and "b" have overlapping values
df = pd.DataFrame({'a': list('aabbccddeeff'), 'b': list('aaaabbbbcccc'),
'c': np.random.randint(5, size=12),
'd': np.random.randint(9, size=12)})
df
df.query('a in b')
# How you'd do it in pure Python
df[df['a'].isin(df['b'])]
df.query('a not in b')
# pure Python
df[~df['a'].isin(df['b'])]
You can combine this with other expressions for very succinct queries:
# rows where cols a and b have overlapping values
# and col c's values are less than col d's
df.query('a in b and c < d')
# pure Python
df[df['b'].isin(df['a']) & (df['c'] < df['d'])]
Note
Note that in
and not in
are evaluated in Python, since numexpr
has no equivalent of this operation. However, only the in
/not in
expression itself is evaluated in vanilla Python. For example, in the
expression
df.query('a in b + c + d')
(b + c + d)
is evaluated by numexpr
and then the in
operation is evaluated in plain Python. In general, any operations that can
be evaluated using numexpr
will be.
Special use of the ==
operator with list
objects#
Comparing a list
of values to a column using ==
/!=
works similarly
to in
/not in
.
df.query('b == ["a", "b", "c"]')
# pure Python
df[df['b'].isin(["a", "b", "c"])]
df.query('c == [1, 2]')
df.query('c != [1, 2]')
# using in/not in
df.query('[1, 2] in c')
df.query('[1, 2] not in c')
# pure Python
df[df['c'].isin([1, 2])]
Returning a view versus a copy#
When setting values in a pandas object, care must be taken to avoid what is called
chained indexing
. Here is an example.
dfmi = pd.DataFrame([list('abcd'),
list('efgh'),
list('ijkl'),
list('mnop')],
columns=pd.MultiIndex.from_product([['one', 'two'],
['first', 'second']]))
dfmi
Compare these two access methods:
dfmi['one']['second']
dfmi.loc[:, ('one', 'second')]
These both yield the same results, so which should you use? It is instructive to understand the order
of operations on these and why method 2 (.loc
) is much preferred over method 1 (chained []
).
dfmi['one']
selects the first level of the columns and returns a DataFrame that is singly-indexed.
Then another Python operation dfmi_with_one['second']
selects the series indexed by 'second'
.
This is indicated by the variable dfmi_with_one
because pandas sees these operations as separate events.
e.g. separate calls to __getitem__
, so it has to treat them as linear operations, they happen one after another.
Contrast this to df.loc[:,('one','second')]
which passes a nested tuple of (slice(None),('one','second'))
to a single call to
__getitem__
. This allows pandas to deal with this as a single entity. Furthermore this order of operations can be significantly
faster, and allows one to index both axes if so desired.
Why does assignment fail when using chained indexing?#
The problem in the previous section is just a performance issue. What’s up with
the SettingWithCopy
warning? We don’t usually throw warnings around when
you do something that might cost a few extra milliseconds!
But it turns out that assigning to the product of chained indexing has inherently unpredictable results. To see this, think about how the Python interpreter executes this code:
value = None
dfmi.loc[:, ('one', 'second')] = value
# becomes
dfmi.loc.__setitem__((slice(None), ('one', 'second')), value)
But this code is handled differently:
dfmi['one']['second'] = value
# becomes
dfmi.__getitem__('one').__setitem__('second', value)
See that __getitem__
in there? Outside of simple cases, it’s very hard to
predict whether it will return a view or a copy (it depends on the memory layout
of the array, about which pandas makes no guarantees), and therefore whether
the __setitem__
will modify dfmi
or a temporary object that gets thrown
out immediately afterward. That’s what SettingWithCopy
is warning you
about!
Note
You may be wondering whether we should be concerned about the loc
property in the first example. But dfmi.loc
is guaranteed to be dfmi
itself with modified indexing behavior, so dfmi.loc.__getitem__
/
dfmi.loc.__setitem__
operate on dfmi
directly. Of course,
dfmi.loc.__getitem__(idx)
may be a view or a copy of dfmi
.
Sometimes a SettingWithCopy
warning will arise at times when there’s no
obvious chained indexing going on. These are the bugs that
SettingWithCopy
is designed to catch! Pandas is probably trying to warn you
that you’ve done this:
def do_something(df):
foo = df[['bar', 'baz']] # Is foo a view? A copy? Nobody knows!
# ... many lines here ...
# We don't know whether this will modify df or not!
foo['quux'] = value
return foo
Yikes!
Evaluation order matters#
When you use chained indexing, the order and type of the indexing operation partially determine whether the result is a slice into the original object, or a copy of the slice.
Pandas has the SettingWithCopyWarning
because assigning to a copy of a
slice is frequently not intentional, but a mistake caused by chained indexing
returning a copy where a slice was expected.
If you would like pandas to be more or less trusting about assignment to a
chained indexing expression, you can set the option
mode.chained_assignment
to one of these values:
'warn'
, the default, means aSettingWithCopyWarning
is printed.'raise'
means pandas will raise aSettingWithCopyException
you have to deal with.None
will suppress the warnings entirely.
dfb = pd.DataFrame({'a': ['one', 'one', 'two',
'three', 'two', 'one', 'six'],
'c': np.arange(7)})
# This will show the SettingWithCopyWarning
# but the frame values will be set
dfb['c'][dfb['a'].str.startswith('o')] = 42
This however is operating on a copy and will not work.
>>> pd.set_option('mode.chained_assignment','warn')
>>> dfb[dfb['a'].str.startswith('o')]['c'] = 42
Traceback (most recent call last)
...
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
A chained assignment can also crop up in setting in a mixed dtype frame.
Note
These setting rules apply to all of .loc/.iloc
.
This is the correct access method:
dfc = pd.DataFrame({'A': ['aaa', 'bbb', 'ccc'], 'B': [1, 2, 3]})
dfc.loc[0, 'A'] = 11
dfc
This can work at times, but it is not guaranteed to, and therefore should be avoided:
dfc = dfc.copy()
dfc['A'][0] = 111
dfc
This will not work at all, and so should be avoided:
>>> pd.set_option('mode.chained_assignment','raise')
>>> dfc.loc[0]['A'] = 1111
Traceback (most recent call last)
...
SettingWithCopyException:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
Warning
The chained assignment warnings / exceptions are aiming to inform the user of a possibly invalid assignment. There may be false positives; situations where a chained assignment is inadvertently reported.
Top-level headers and the TOC#
Your right table of contents will behave slightly differently depending on whether your page has one top-level header, or multiple top-level headers. See below for more information.
An example with multiple top-level headers#
If a page has multiple top-level headers on it, then the in-page Table of Contents will show each top-level header. On this page, there are multiple top-level headers. As a result, the top-level headers all appear in the right Table of Contents. Here’s an example of a page structure with multiple top-level headers:
My first header
===============
My sub-header
-------------
My second header
================
My second sub-header
--------------------
And here’s a second-level header#
Notice how it is nested underneath “Top-level header 2” in the TOC.
An example with a single top-level header#
If the page only has a single top-level header, it is assumed to be the page title, and only the headers underneath the top-level header will be used for the right Table of Contents.
On most pages in this documentation, only a single top-level header is used. For example, they have a page structure like:
My title
========
My header
---------
My second header
----------------
Section to show off pages with many sub-pages#
To create an additional level of nesting in the sidebar, construct a
nested toctree
:
Sub-page 1#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-page 2#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Section with sub-sub-pages#
To create an additional level of nesting in the sidebar, construct a
nested toctree
:
Sub-sub-page 1#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 2#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 3#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 4#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 5#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 6#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 7#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 8#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 9#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 10#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 11#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 12#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 13#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 14#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 15#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 16#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 17#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 18#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 19#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.
Sub-sub-page 20#
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec lorem neque, interdum in ipsum nec, finibus dictum velit. Ut eu efficitur arcu, id aliquam erat. In sit amet diam gravida, imperdiet tellus eu, gravida nisl. Praesent aliquet odio eget libero elementum, quis rhoncus tellus tincidunt. Suspendisse quis volutpat ipsum. Sed lobortis scelerisque tristique. Aenean condimentum risus tellus, quis accumsan ipsum laoreet ut. Integer porttitor maximus suscipit. Mauris in posuere sapien. Aliquam accumsan feugiat ligula, nec fringilla libero commodo sed. Proin et erat pharetra.