In order to index the web properly, corbia spider uses domain based templates.

## Domain declaration

For each domain, a custom python function will be used in order to extract the relevant information and store it as JSON before further processing.

New domains are added to `Domain` class:

```python
class Domain:

def __init__(self, domain, html, url):

	self.domain = domain
	self.soup = BeautifulSoup(html, 'html.parser')
	self.url = url
	self.scrapers = {
		"www.azlyrics.com" : self.scrape_azlyrics,
		"www.monde-diplomatique.fr" : self.scrape_diplo,
		"www.amnesty.org" : self.scrape_amnesty,
		"www.vindefrance.com" : self.scrape_vdf,
		"www.tasteofcinema.com" : self.scrape_taste_of_cinema,
		"www.blast-info.fr" : self.scrape_blast,
		"www.byredo.com" : self.scrape_byredo
	}
```

Then, the function is added to the class as `def scrape_domain(self)`.

## Domain based scrapping function

The function must **ALWAYS** return a JSON with a `"success"` key set to either `True` or `False`. 

To make sure the spider doesn't stop working, the scrapping function must be wrapped inside a `try` / `except` block:

```python
def scrape_domain(self):
	try:
		data = {
			"success" : True
		}
		return json.dumps(data, indent=4, ensure_ascii=False)
	except Exception as e:
		return json.dumps({"success": False})
```