API Documentation¶

Providers¶

class micawber.providers.Provider(endpoint[, timeout=3.0[, user_agent=None[, **kwargs]]])¶

The Provider object is responsible for retrieving metadata about a given URL. It implements a method called request(), which takes a URL and any parameters, which it sends off to an endpoint. The endpoint should return a JSON dictionary containing metadata about the resource, which is returned to the caller.

Parameters:

endpoint – the API endpoint which should return information about requested links
timeout (float) – socket timeout, in seconds, for requests to the endpoint.
user_agent (str) – value sent in the User-Agent header.
kwargs – any additional url parameters to send to the endpoint on each request, used for providing defaults. An example use-case might be for providing an API key on each request.

request(url, **extra_params)¶

Retrieve information about the given url. By default, will make a HTTP GET request to the endpoint. The url will be sent to the endpoint, along with any parameters specified in the extra_params and those parameters specified when the class was instantiated.

Raises ProviderException when the request fails. The subclass says why, so a caller can cache a 404, report a 401, and retry a timeout:

ProviderHTTPException for a non-2xx response, with the code on .status
ProviderTimeoutException when the endpoint does not answer in time
InvalidResponseException when the body is not a JSON object
ProviderException itself for anything else, such as a refused connection or a bad charset

The underlying error is on __cause__.

Parameters:

url – URL to retrieve metadata for
extra_params – additional parameters to pass to the endpoint, for example a maxwidth or an API key. A maxwidth given without a maxheight implies a maxheight of 16/9 the width, tall enough that a portrait embed is not squashed.

Return type:

a dictionary of JSON data

class micawber.providers.ProviderRegistry([cache=None[, max_workers=None[, negative_ttl=300]]])¶

A registry for encapsulating a group of Provider instances, with optional caching support.

Handles matching regular expressions to providers. URLs are sent to the registry via its request() method, it checks to see if it has a provider that matches the URL, and if so, requests the metadata from the provider instance.

Exposes methods for parsing various types of text (including HTML), and either rendering oembed media inline or extracting embeddable links.

Parameters:

cache – the cache simply needs to implement two methods, .get(key) and .set(key, value).
max_workers (int) – fetch the URLs in a document concurrently, using a thread pool of this size. Unset, requests are made one at a time.
negative_ttl (int) – seconds to remember that a provider failed for a URL, so a dead link is not re-requested on every render. Within that window the request raises ProviderException without touching the provider. 0 disables it. Needs nothing from the cache beyond get and set.

register(regex, provider[, skip_invalid=False])¶

Register the provider with the following regex. The regex is compiled once here rather than on every lookup.

Example:

registry = ProviderRegistry()
registry.register(
    'https://\S*.youtu(\.be|be\.com)/watch\S*',
    Provider('https://www.youtube.com/oembed'),
)

Parameters:

regex – a regex for matching URLs of a given type
provider – a Provider instance
skip_invalid (bool) – log a warning and skip the provider when the regex will not compile, rather than raising re.error. Used when registering patterns from a third-party provider list, where one bad pattern should not cost the whole list. Patterns you write yourself should raise.

request(url, **extra_params)¶

Retrieve information about the given url if it matches a regex in the instance’s registry. If no provider matches the URL, a ProviderException is thrown, otherwise the URL and parameters are dispatched to the matching provider’s Provider.request() method.

If a cache was specified, the resulting metadata will be cached.

Parameters:

url – URL to retrieve metadata for
extra_params – additional parameters to pass to the endpoint, for example a maxwidth or an API key.

Return type:

a dictionary of JSON data

request_many(urls, **extra_params)¶

Retrieve information about each url, concurrently when max_workers is set. Returns a dict of url to metadata holding only the urls that resolved. Duplicates are requested once. This is what the parsers call, once per document.

Parameters:

urls – an iterable of URLs
extra_params – additional parameters to pass to each provider

Return type:

dict

parse_text_full(text[, urlize_all=True[, handler=full_handler[, urlize_params=None[, **params]]]])¶

Parse a block of text, converting all links by passing them to the given handler. Links contained within a block of text (i.e. not on their own line) will be handled as well.

Example input and output:

IN: 'this is a pic http://example.com/some-pic/'
OUT: 'this is a pic <a href="http://example.com/some-pic/"><img src="http://example.com/media/some-pic.jpg" /></a>'

Parameters:

text (str) – a string to parse
urlize_all (bool) – convert unmatched urls into links
handler – function to use to convert metadata back into a string representation
urlize_params (dict) – keyword arguments to be used to construct a link when a provider is not found and urlize is enabled.
params – any additional parameters to use when requesting metadata, i.e. a maxwidth or maxheight.

parse_text(text[, urlize_all=True[, handler=full_handler[, block_handler=inline_handler[, urlize_params=None[, **params]]]]])¶

Very similar to parse_text_full() except URLs on their own line are rendered using the given handler, whereas URLs within blocks of text are passed to the block_handler. The default behavior renders full content for URLs on their own line (e.g. a video player), whereas URLs within text are rendered simply as links so as not to disrupt the flow of text.

URLs on their own line are converted into full representations
URLs within blocks of text are converted into clickable links

Parameters:

text (str) – a string to parse
urlize_all (bool) – convert unmatched urls into links
handler – function to use to convert links found on their own line
block_handler – function to use to convert links found within blocks of text
urlize_params (dict) – keyword arguments to be used to construct a link when a provider is not found and urlize is enabled.
params – any additional parameters to use when requesting metadata, i.e. a maxwidth or maxheight.

parse_html(html[, urlize_all=True[, handler=full_handler[, block_handler=inline_handler[, urlize_params=None[, **params]]]]])¶

Parse HTML intelligently, rendering items on their own within block elements as full content (e.g. a video player), whereas URLs within text are passed to the block_handler which by default will render a simple link. URLs that are already enclosed within a <a> tag are skipped over.

URLs that are already within <a> tags are passed over
URLs on their own in block tags are converted into full representations
URLs interspersed with text are converted into clickable links

Note

requires BeautifulSoup or beautifulsoup4

Parameters:

html (str) – a string of HTML to parse
urlize_all (bool) – convert unmatched urls into links
handler – function to use to convert links found on their own within a block element
block_handler – function to use to convert links found within blocks of text
urlize_params (dict) – keyword arguments to be used to construct a link when a provider is not found and urlize is enabled.
params – any additional parameters to use when requesting metadata, i.e. a maxwidth or maxheight.

extract(text, **params)¶

Extract all URLs from a block of text, and additionally get any metadata for URLs we have providers for.

Parameters:

text (str) – a string to parse
params – any additional parameters to use when requesting metadata, i.e. a maxwidth or maxheight.

Return type:

returns a 2-tuple containing a list of all URLs and a dict keyed by URL containing any metadata. If a provider was not found for a URL it is not listed in the dictionary.

extract_html(html, **params)¶

Extract all URLs from an HTML string, and additionally get any metadata for URLs we have providers for. extract() but for HTML.

Note

URLs within <a> tags will not be included.

Parameters:

html (str) – a string to parse
params – any additional parameters to use when requesting metadata, i.e. a maxwidth or maxheight.

Return type:

returns a 2-tuple containing a list of all URLs and a dict keyed by URL containing any metadata. If a provider was not found for a URL it is not listed in the dictionary.

micawber.providers.bootstrap_basic([cache=None[, registry=None[, max_workers=None]]])¶

Create a ProviderRegistry with the major platforms registered: YouTube, Vimeo, X, TikTok, Instagram, Facebook, Reddit, Spotify, Bluesky, SoundCloud, Flickr, Giphy, Pinterest, Tumblr, Dailymotion, Apple Podcasts, Apple Music and WordPress.com. Every endpoint was verified live against a real URL. No request is made by this function itself. For the long tail, use bootstrap_oembed().

Parameters:

cache – an object that implements simple get and set
registry – a ProviderRegistry instance, which will be updated with the list of supported providers. If not specified, an empty ProviderRegistry will be used.
max_workers (int) – thread pool size for the registry this function creates, as ProviderRegistry takes it. Ignored when registry is given, since that registry already has its own.

Return type:

a ProviderRegistry with a handful of providers registered

micawber.providers.bootstrap_oembed([cache=None[, registry=None[, refresh=False[, timeout=3.0[, providers_file=None[, max_workers=None[, **kwargs]]]]]])¶

Create a ProviderRegistry and register as many providers as are described in the oembed.com providers list.

By default the list is read from the copy shipped with micawber, so no request is made. To use a newer list without waiting for a release, either pass refresh=True to fetch it live, or regenerate a local copy and point providers_file at it:

python -m micawber my_providers.json

Run with no argument, that command updates the copy inside the installed package instead. The same thing is available as refresh_providers().

Parameters:

cache – an object that implements simple get and set
registry – a ProviderRegistry instance, which will be updated with the list of supported providers. If not specified, an empty ProviderRegistry will be used.
max_workers (int) – thread pool size for the registry this function creates, as ProviderRegistry takes it. Ignored when registry is given, since that registry already has its own.
refresh (bool) – fetch the list from oembed.com instead of reading a file. The result is stored in cache if one is given.
providers_file (str) – path to a provider list to read instead of the shipped one. Ignored when refresh is set.
timeout (float) – socket timeout, in seconds, applied to the provider-list request made by this function and to the providers it registers. Without a bounded default, a provider-list server that accepts the connection and never replies blocks forever.
kwargs – any default keyword arguments to use with providers

Return type:

a ProviderRegistry with support for noembed

micawber.providers.refresh_providers([path])¶: Download the current oembed.com provider list and write it to path, which defaults to the copy inside the installed package. The download is checked to be a JSON array before anything is overwritten. Returns the path written.

micawber.providers.bootstrap_embedly([cache=None[, registry=None[, refresh=False[, timeout=3.0[, max_workers=None[, **kwargs]]]]])¶

Create a ProviderRegistry and register as many providers as are supported by embed.ly. Valid services are fetched from http://api.embed.ly/1/services/python and parsed then registered.

Note

This function makes a request over the internet whenever it is called.

Parameters:

cache – an object that implements simple get and set
registry – a ProviderRegistry instance, which will be updated with the list of supported providers. If not specified, an empty ProviderRegistry will be used.
max_workers (int) – thread pool size for the registry this function creates, as ProviderRegistry takes it. Ignored when registry is given, since that registry already has its own.
refresh (bool) – force refreshing the provider data rather than attempting to load it from cache (if cache is used).
timeout (float) – socket timeout, in seconds, applied to the provider-list request made by this function and to the providers it registers. Without a bounded default, a provider-list server that accepts the connection and never replies blocks forever.
kwargs – any default keyword arguments to use with providers, useful for specifying your API key

Return type:

a ProviderRegistry with support for embed.ly

# if you have an API key, you can specify that here
pr = bootstrap_embedly(key='my-embedly-key')
pr.request('https://www.youtube.com/watch?v=54XHDUOHuzU')

micawber.providers.bootstrap_noembed([cache=None[, registry=None[, refresh=False[, timeout=3.0[, max_workers=None[, **kwargs]]]]])¶

Create a ProviderRegistry and register as many providers as are supported by noembed.com. Valid services are fetched from https://noembed.com/providers and parsed then registered.

Note

This function makes a request over the internet whenever it is called.

Parameters:

cache – an object that implements simple get and set
registry – a ProviderRegistry instance, which will be updated with the list of supported providers. If not specified, an empty ProviderRegistry will be used.
max_workers (int) – thread pool size for the registry this function creates, as ProviderRegistry takes it. Ignored when registry is given, since that registry already has its own.
refresh (bool) – force refreshing the provider data rather than attempting to load it from cache (if cache is used).
timeout (float) – socket timeout, in seconds, applied to the provider-list request made by this function and to the providers it registers. Without a bounded default, a provider-list server that accepts the connection and never replies blocks forever.
kwargs – any default keyword arguments to use with providers, useful for passing the nowrap option to noembed.

Return type:

a ProviderRegistry with support for noembed

# if you have an API key, you can specify that here
pr = bootstrap_noembed(nowrap=1)
pr.request('https://www.youtube.com/watch?v=54XHDUOHuzU')

micawber.providers.bootstrap_iframely([cache=None[, registry=None[, max_workers=None[, **kwargs]]]])¶

Create a ProviderRegistry that routes requests through iframely, a commercial oEmbed proxy supporting roughly 1900 domains. Iframely recommends sending all URLs to its API rather than matching against a list of supported providers, so a single catch-all pattern is registered. Unlike the other schema-based helpers, this function does not make a request over the internet when called.

An iframely API key is required, passed as either api_key or key (the md5 hexdigest of the api key).

Parameters:

cache – an object that implements simple get and set
registry – a ProviderRegistry instance, which will be updated with the list of supported providers. If not specified, an empty ProviderRegistry will be used.
max_workers (int) – thread pool size for the registry this function creates, as ProviderRegistry takes it. Ignored when registry is given, since that registry already has its own.
kwargs – any default keyword arguments to use with providers - must include api_key or key.

Return type:

a ProviderRegistry with support for iframely

pr = bootstrap_iframely(api_key='my-iframely-key')
pr.request('https://www.youtube.com/watch?v=54XHDUOHuzU')

Note

Because providers registered later take precedence, passing an existing registry to this function will cause the catch-all iframely provider to shadow any previously-registered providers.

Cache¶

class micawber.cache.Cache([timeout=None[, max_size=1024]])¶

A reference implementation for the cache interface used by the ProviderRegistry.

Parameters:

timeout (int) – seconds a value stays fresh. Unset, values never expire and are only dropped to stay within max_size.
max_size (int) – how many entries to keep. Once full, the least-recently-used entry is evicted. 0 or None is unbounded.

from micawber import Cache, bootstrap_oembed
cache = Cache()  # Simple in-memory cache.

# Entries expire after an hour, and at most 4096 are kept.
cache = Cache(timeout=3600, max_size=4096)

# Now our oembed provider will cache the responses for each URL we
# request, which can provide a significant speedup.
pr = bootstrap_oembed(cache=cache)

get(key)¶: Retrieve the key from the cache or None if not present or expired.

set(key, value[, timeout=None])¶

Set the cache key key to the given value.

Parameters:: timeout (int) – expiration for this value, overriding the cache default.

class micawber.cache.PickleCache([filename='cache.db'[, timeout=None[, max_size=None]]])¶

A cache that uses pickle to store data. Accepts the same timeout and max_size as Cache, and by default keeps every entry.

Note

To use this cache class be sure to call load() when initializing your cache and save() before your app terminates to persist cached data.

load()¶: Load the pickled data into memory

save()¶: Store the internal cache to an external file

class micawber.cache.RedisCache([namespace='micawber'[, timeout=None[, **conn]]])¶

A cache that uses Redis to store data

Note

requires the redis-py library, pip install redis

Parameters:

namespace – prefix for cache keys
timeout (int) – expiration timeout in seconds (optional), applied to every value written. set() can override it per-value.
conn – keyword arguments to pass when initializing redis connection

API Documentation¶

Providers¶

Cache¶

Table of Contents

Previous topic

This Page