diff options
| author | Lars Wirzenius <liw@liw.fi> | 2017-11-13 11:43:37 +0100 |
|---|---|---|
| committer | Lars Wirzenius <liw@liw.fi> | 2017-11-14 12:06:33 +0100 |
| commit | 5bca14e72c204b31424069ad0090aefa76071e8c (patch) | |
| tree | ce00b14ed66436d7175ce04b0ce48f1b89e97c3f | |
| parent | 607f62ae6bc8f684095e447569cff27ba7c8a5dd (diff) | |
| download | qvisqve-5bca14e72c204b31424069ad0090aefa76071e8c.tar.gz | |
Fix: start Salami
This is based on Qvarn, because the implementation is so similar to
hat Salami needs. This commit drops the unwanted bits of Qvarn and
changes things to be Salami instead.
This commit only introduces the /version endpoint.
72 files changed, 229 insertions, 7790 deletions
@@ -1,41 +1,9 @@ -NEWS for Qvarn (JSONB) +NEWS for Salami ============================================================================= -This file has release notes for Qvarn (the JSONB version). +This file has release notes for Salami -Version 0.86+git, not yet released +Version 0.0+git, not yet released ---------------------------------- -Bug fixes: - -* Listeners for all resource types were listed for all resource types. - Thus, `GET /foos/listners` would list listeners also for `bar` - resources, not just `foo`. This has been fixed. Bug reported by - Kaius, fixed by Lars. - -* Mantas reported that if a resource type or resource had two fields - with the same type but values that were not possible to compare, - such as a string and an integer, Qvarn would crash. This is now - fixed. - -* Saulius Žemaitaitis reported that updating a resource type - specification YAML file and restarting Qvarn wouldn't make the new - version of the resource type availables. Fixed now. - -Version 0.86, released 2017-10-10 ----------------------------------- - -* All old API tests (`test-api`) pass. Qvarn (JSONB) is functionally - complete now, and API compatible with the old Qvarn. It may not be - performant enough yet. - -Version 0.84, released 2017-09-27 ------------------------------------------------------------------------------ - -* This is the first release of Qvarn (JSONB), and has been rewritten - from scratch to use the Postgres JSONB data type for storing - resources. It is NOT yet ready for production use. This release is - done so we can test Ansible modifications, run benchmarks, and other - such things. - - Do not use this release. +First release. @@ -1,38 +1,13 @@ -README for qvarn-jsonb +README for Salami ============================================================================= -This is the JSONB-using version of [Qvarn][], a web application to -manage structured data via a RESTful HTTP JSON API. It is primarily -meant for personal data, and is meant to make it easier to comply with -the EU [General Data Protection Regulation][]. - -[Qvarn]: http://qvarn.org/ -[General Data Protection Regulation]: https://en.wikipedia.org/wiki/General_Data_Protection_Regulation - -History and background ------------------------------------------------------------------------------ - -Qvarn started as an internal project in 2015 at Suomen Tilaajavastuu -(https://www.tilaajavastuu.fi/en/), a company that provides various -reproting services to the Finnish construction industry. Later that -year the Swedish Construction Federation -(https://www.sverigesbyggindustrier.se/english) adopted Qvarn as a -platform for ID06 identity card services in Sweden. From the start, a -goal for Qvarn has been to make it easy to comply with the EU General -Data Protection Regulation. - -In 2016 Qvarn spun off to its own company, QvarnLabs Ab, which -provides consulting, development, and support services around Qvarn. - Legalese ----------------------------------------------------------------------------- Qvarn in its entirety is copyright by its authorss, and released under the GNU Affero General Public Licence, version 3, or later. - Qvarn-JSONB, a RESTful HTTP API for storing structured personal information - Copyright (C) 2015, 2016 Suomen Tilaajavastuu Oy - Copyright (C) 2017 QvarnLabs Ab + Salami Copyright (C) 2017 Lars Wirzenius This program is free software: you can redistribute it and/or modify @@ -17,22 +17,30 @@ set -eu +title() +{ + printf '\n%s\n' "$@" + for i in $(seq 77) + do + printf '%c' - + done + printf '\n' +} + run_yarns() { yarn yarns/*.yarn \ -s yarns/lib.py --shell python2 --shell-arg '' --cd-datadir "$@" } +title "Unit tests" python3 -m CoverageTestRunner --ignore-missing-from=without-tests -pep8 qvarn -pylint3 -j0 --rcfile pylint.conf qvarn - -if [ "${QVARN_POSTGRES:-no}" = no ] -then - # Run yarns without Postgres. For speed. - run_yarns "$@" -else - scripts/pgempty - run_yarns --env "QVARN_POSTGRES=$QVARN_POSTGRES" "$@" -fi +title "Code style" +pycodestyle salami + +title "Static checking" +pylint3 -j0 --rcfile pylint.conf salami + +title OK +echo All tests pass. diff --git a/debian/changelog b/debian/changelog index df2aea5..1f2cb16 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,22 +1,4 @@ -qvarn-jsonb (0.86+git-1) UNRELEASED; urgency=medium - - * New upstream version. - - -- Lars Wirzenius <liw@liw.fi> Tue, 10 Oct 2017 11:58:51 +0300 - -qvarn-jsonb (0.86-1) unstable; urgency=medium - - * New upstream version. - - -- Lars Wirzenius <liw@liw.fi> Tue, 10 Oct 2017 11:58:48 +0300 - -qvarn-jsonb (0.85-1) unstable; urgency=medium - - * New upstream version. - - -- Lars Wirzenius <liw@liw.fi> Wed, 27 Sep 2017 19:21:33 +0300 - -qvarn-jsonb (0.84-1) unstable; urgency=medium +salami (0.1-1) unstable; urgency=medium * Initial packaging. This is not intended to be uploaded to Debian, so no closing of an ITP bug. diff --git a/debian/control b/debian/control index 4b3ce7b..90c2f98 100644 --- a/debian/control +++ b/debian/control @@ -1,4 +1,4 @@ -Source: qvarn-jsonb +Source: salami Maintainer: QvarnLabs <info@qvarnlabs.com> Uploaders: Lars Wirzenius <liw@qvarnlabs.com> Section: web @@ -12,11 +12,9 @@ Build-Depends: debhelper (>= 9), python3-all, python3-apifw X-Python3-Version: >= 3.5 -Package: qvarn-jsonb +Package: salami Architecture: all Depends: ${python3:Depends}, ${misc:Depends}, python3 (>= 3.5) -Description: web service backend for structured data storage - This is a backend service for storing structured (JSON) and some - associated binary data using a REST-ful HTTP API and OpenID Connect - authentication. +Description: a thing + This is a thing. diff --git a/debian/copyright b/debian/copyright index 668a192..687e273 100644 --- a/debian/copyright +++ b/debian/copyright @@ -1,5 +1,5 @@ Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ -Upstream-Name: qvarn +Upstream-Name: salami Files: * Copyright: 2017, Lars Wirzenius diff --git a/debian/qvarn-jsonb.install b/debian/qvarn-jsonb.install deleted file mode 100644 index a8fe43f..0000000 --- a/debian/qvarn-jsonb.install +++ /dev/null @@ -1,2 +0,0 @@ -resource_type/*.yaml etc/qvarn/resource_type -start_qvarn usr/bin @@ -1,54 +0,0 @@ -#!/usr/bin/python3 - - -import qvarn - - -args = { - 'host': 'localhost', - 'port': 5432, - 'database': 'qvarn', - 'user': 'qvarn', - 'password': 'pass', - 'min_conn': 1, - 'max_conn': 1, -} - - -sql = qvarn.PostgresAdapter() -sql.connect(**args) - -pos = qvarn.PostgresObjectStore(sql) -pos.create_store(obj_id=str, revision=str) - - -objs = [ - { - 'foo': 'bar', - 'id': '1', - 'revision': '1.1', - }, - { - 'foo': 'yo', - 'id': '2', - 'revision': '2.1', - }, -] -for obj in objs: - pos.create_object(obj, obj_id=obj['id'], revision=obj['revision']) - -print('finding') -# cond = qvarn.Equal('foo', 'bar') -cond = qvarn.Yes() -objs = pos.get_objects(obj_id=obj['id']) -print(repr(objs)) -for obj in objs: - print('obj: ', repr(obj)) -print('done finding') - -print('finding keys') -cond = qvarn.Equal('foo', 'bar') -objs = pos.find_objects(cond) -for obj in objs: - print('keys: ', repr(obj)) -print('done finding keys') diff --git a/qvarn/__init__.py b/qvarn/__init__.py deleted file mode 100644 index 7d35b1a..0000000 --- a/qvarn/__init__.py +++ /dev/null @@ -1,114 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -from .version import __version__, __version_info__ -from .logging import log, setup_logging -from .idgen import ResourceIdGenerator -from .schema import schema -from .resource_type import ( - ResourceType, - load_resource_types, - add_missing_fields, -) -from .sql import ( - PostgresAdapter, - quote, - placeholder, - get_unique_name, - All, - Contains, - Equal, - GreaterThan, - GreaterOrEqual, - LessThan, - LessOrEqual, - NotEqual, - ResourceTypeIs, - Startswith, - Yes, -) -from .sql_select import sql_select, flatten - -from .objstore import ( - ObjectStoreInterface, - MemoryObjectStore, - PostgresObjectStore, - KeyCollision, - UnknownKey, - WrongKeyType, - KeyValueError, - NoSuchObject, - BlobKeyCollision, - flatten_object, -) - -from .validator import ( - Validator, - ValidationError, - NotADict, - NoType, - HasId, - HasRevision, - NoId, - NoRevision, - UnknownField, - UnknownSubpath, - WrongType, -) - -from .search_parser import ( - SearchParser, - SearchParameters, - SearchParserError, - NeedSortOperator, -) - -from .collection import ( - CollectionAPI, - NoSearchCriteria, - NoSuchResource, - UnknownSearchField, - WrongRevision, -) - -from .responses import ( - bad_request_response, - conflict_response, - created_response, - need_sort_response, - no_such_resource_response, - ok_response, - search_parser_error_response, - unknown_search_field_response, -) - -from .api_errors import ( - IdMismatch, - NoSuchResourceType, - NotJson, - TooManyResources, - TooManyResourceTypes, -) - -from .router import Router -from .file_router import FileRouter -from .notification_router import NotificationRouter -from .resource_router import ResourceRouter -from .subresource_router import SubresourceRouter -from .version_router import VersionRouter -from .timestamp import get_current_timestamp - -from .api import QvarnAPI diff --git a/qvarn/api.py b/qvarn/api.py deleted file mode 100644 index aea0887..0000000 --- a/qvarn/api.py +++ /dev/null @@ -1,187 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class QvarnAPI: - - def __init__(self): - self._store = None - self._validator = qvarn.Validator() - self._baseurl = None - self._rt_coll = None - - def set_base_url(self, baseurl): # pragma: no cover - self._baseurl = baseurl - - def set_object_store(self, store): - self._store = store - self._store.create_store(obj_id=str, subpath=str) - - def add_resource_type(self, rt): - path = rt.get_path() - keys = { - 'obj_id': rt.get_type(), - 'subpath': '', - } - self._store.remove_objects(**keys) - - obj = { - 'id': rt.get_type(), - 'type': 'resource_type', - 'path': path, - 'spec': rt.as_dict(), - } - self._store.create_object(obj, **keys, auxtable=True) - - def get_resource_type(self, path): - objs = self._get_resource_type_given_path(path) - if len(objs) == 0: - qvarn.log.log( - 'error', - msg_text='There is no resource type for path', - path=path) - raise qvarn.NoSuchResourceType(path) - elif len(objs) > 1: # pragma: no cover - qvarn.log.log( - 'error', - msg_text='There are more than one resource types for path', - path=path, - objs=objs) - raise qvarn.TooManyResourceTypes(path) - rt = qvarn.ResourceType() - rt.from_spec(objs[0]['spec']) - return rt - - def _get_resource_type_given_path(self, path): - cond = qvarn.All( - qvarn.Equal('path', path), - qvarn.Equal('type', 'resource_type'), - ) - results = self._store.find_objects(cond) - return [obj for _, obj in results] - - def get_listener_resource_type(self): - return self._get_resource_type_given_type('listener') - - def get_notification_resource_type(self): # pragma: no cover - return self._get_resource_type_given_type('notification') - - def _get_resource_type_given_type(self, type_name): - cond = qvarn.All( - qvarn.Equal('id', type_name), - qvarn.Equal('type', 'resource_type'), - ) - results = self._store.find_objects(cond) - objs = [obj for _, obj in results] - - if len(objs) == 0: # pragma: no cover - raise qvarn.NoSuchResourceType(type_name) - elif len(objs) > 1: # pragma: no cover - raise qvarn.TooManyResourceTypes(type_name) - - rt = qvarn.ResourceType() - rt.from_spec(objs[0]['spec']) - return rt - - def find_missing_route(self, path): - qvarn.log.log('info', msg_text='find_missing_route', path=path) - - if path == '/version': - qvarn.log.log('info', msg_text='Add /version route') - v = qvarn.VersionRouter() - return v.get_routes() - - try: - rt = self.get_resource_type(path) - except qvarn.NoSuchResourceType: - qvarn.log.log('warning', msg_text='No such route', path=path) - return [] - - routes = self.resource_routes(path, rt) - qvarn.log.log('info', msg_text='Found missing routes', routes=routes) - return routes - - def resource_routes(self, path, rt): # pragma: no cover - coll = qvarn.CollectionAPI() - coll.set_object_store(self._store) - coll.set_resource_type(rt) - - router = qvarn.ResourceRouter() - router.set_baseurl(self._baseurl) - router.set_collection(coll) - router.set_notifier(self.notify) - routes = router.get_routes() - - files = rt.get_files() - for subpath in rt.get_subpaths(): - if subpath not in files: - sub_router = qvarn.SubresourceRouter() - sub_router.set_subpath(subpath) - sub_router.set_parent_collection(coll) - more = sub_router.get_routes() - else: - file_router = qvarn.FileRouter() - file_router.set_subpath(subpath) - file_router.set_object_store(self._store) - file_router.set_parent_collection(coll) - more = file_router.get_routes() - routes.extend(more) - - listener_rt = self.get_listener_resource_type() - notif_router = qvarn.NotificationRouter() - notif_router.set_baseurl(self._baseurl) - notif_router.set_parent_collection(coll) - notif_router.set_object_store(self._store, listener_rt) - routes.extend(notif_router.get_routes()) - - return routes - - def notify(self, rid, rrev, change): # pragma: no cover - rt = self.get_notification_resource_type() - notifs = qvarn.CollectionAPI() - notifs.set_object_store(self._store) - notifs.set_resource_type(rt) - obj = { - 'type': 'notification', - 'resource_id': rid, - 'resource_revision': rrev, - 'resource_change': change, - 'timestamp': qvarn.get_current_timestamp(), - } - for listener in self.find_listeners(rid, change): - obj['listener_id'] = listener['id'] - qvarn.log.log( - 'info', msg_text='Notify listener of change', - notification=obj) - notifs.post(obj) - - def find_listeners(self, rid, change): # pragma: no cover - cond = qvarn.Equal('type', 'listener') - pairs = self._store.find_objects(cond) - for _, obj in pairs: - if self.listener_matches(obj, rid, change): - yield obj - - def listener_matches(self, obj, rid, change): # pragma: no cover - if change == 'created' and obj.get('notify_of_new'): - return True - if change != 'created' and obj.get('listen_on_all'): - return True - if rid in obj.get('listen_on', []): - return True - return False diff --git a/qvarn/api_errors.py b/qvarn/api_errors.py deleted file mode 100644 index 368ddc5..0000000 --- a/qvarn/api_errors.py +++ /dev/null @@ -1,45 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -class NoSuchResourceType(Exception): # pragma: no cover - - def __init__(self, path): - super().__init__('No resource type for path {}'.format(path)) - - -class TooManyResourceTypes(Exception): # pragma: no cover - - def __init__(self, path): - super().__init__('Too many resource types for path {}'.format(path)) - - -class TooManyResources(Exception): # pragma: no cover - - def __init__(self, resource_id): - super().__init__('Too many resources with id {}'.format(resource_id)) - - -class NotJson(Exception): # pragma: no cover - - def __init__(self, ct): - super().__init__('Was expecting application/json, not {}'.format(ct)) - - -class IdMismatch(Exception): # pragma: no cover - - def __init__(self, obj_id, id_from_path): - super().__init__( - 'Resource has id {} but path says {}'.format(obj_id, id_from_path)) diff --git a/qvarn/api_tests.py b/qvarn/api_tests.py deleted file mode 100644 index b6f8929..0000000 --- a/qvarn/api_tests.py +++ /dev/null @@ -1,176 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import copy -import os -import unittest - - -import qvarn - - -class QvarnAPITests(unittest.TestCase): - - def test_returns_routes_for_version_path(self): - api = qvarn.QvarnAPI() - self.assertNotEqual(api.find_missing_route('/version'), []) - - def test_returns_no_routes_for_unknown_resource_type(self): - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - self.assertEqual(api.find_missing_route('/subjects'), []) - - def test_returns_routes_for_known_resource_type(self): - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'id': '', - 'revision': '', - 'name': '', - }, - 'subpaths': { - 'sub': { - 'prototype': { - 'subfoo': '', - }, - }, - }, - }, - ], - } - - rt = qvarn.ResourceType() - rt.from_spec(spec) - - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - api.add_resource_type(rt) - api.set_base_url('https://qvarn.example.com') - - dirname = os.path.dirname(qvarn.__file__) - dirname = os.path.join(dirname, '../resource_type') - resource_types = qvarn.load_resource_types(dirname) - for rt in resource_types: - api.add_resource_type(rt) - - self.assertNotEqual(api.find_missing_route('/subjects'), []) - - def test_get_resource_type_raises_error_for_unknown_path(self): - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - with self.assertRaises(qvarn.NoSuchResourceType): - api.get_resource_type('/subjects') - - def test_get_resource_type_returns_it_when_it_is_known(self): - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'id': '', - 'revision': '', - 'name': '', - }, - }, - ], - } - - rt = qvarn.ResourceType() - rt.from_spec(spec) - - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - api.add_resource_type(rt) - rt2 = api.get_resource_type(spec['path']) - self.assertTrue(isinstance(rt2, qvarn.ResourceType)) - self.assertEqual(rt2.as_dict(), spec) - - def test_add_resource_type_is_ok_adding_type_again(self): - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'id': '', - 'revision': '', - 'name': '', - }, - }, - ], - } - - rt = qvarn.ResourceType() - rt.from_spec(spec) - - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - api.add_resource_type(rt) - self.assertEqual(api.add_resource_type(rt), None) - - def test_updating_resource_type_with_new_version_works(self): - spec1 = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'id': '', - 'revision': '', - 'name': '', - }, - }, - ], - } - - spec2 = copy.deepcopy(spec1) - spec2['versions'].append({ - 'version': 'v1', - 'prototype': { - 'id': '', - 'revision': '', - 'name': '', - 'newfield': '', - }, - }) - - store = qvarn.MemoryObjectStore() - api = qvarn.QvarnAPI() - api.set_object_store(store) - - rt1 = qvarn.ResourceType() - rt1.from_spec(spec1) - api.add_resource_type(rt1) - - rt2 = qvarn.ResourceType() - rt2.from_spec(spec2) - api.add_resource_type(rt2) - - rt = api.get_resource_type(spec1['path']) - self.assertEqual(rt.as_dict(), spec2) diff --git a/qvarn/backend.py b/qvarn/backend.py deleted file mode 100644 index 4a3eab8..0000000 --- a/qvarn/backend.py +++ /dev/null @@ -1,153 +0,0 @@ -#!/usr/bin/python3 -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -# This is a Qvarn backend program that serves all the API requests. It -# is based on apifw, which takes care of routing, logging, and -# authorization. - - -import os - -import yaml - -import apifw -import qvarn -import slog - - -DEFAULT_CONFIG_FILE = '/dev/null' - - -def dict_logger(log, stack_info=None): - qvarn.log.log(exc_info=stack_info, **log) - - -def read_config(filename): - with open(filename) as f: - return yaml.safe_load(f) - - -def check_config(cfg): - for key in cfg: - if cfg[key] is None: - raise Exception('Configration %s should not be None' % key) - - -_counter = slog.Counter() - - -def counter(): - new_context = 'HTTP transaction {}'.format(_counter.increment()) - qvarn.log.set_context(new_context) - - -default_config = { - 'baseurl': 'https://unconfigured-base-url/', - 'token-public-key': None, - 'token-audience': None, - 'token-issuer': None, - 'log': [], - 'resource-type-dir': None, - 'memory-database': True, - 'database': { - 'host': None, - 'port': 5432, - 'database': None, - 'user': None, - 'min_conn': 1, - 'max_conn': 1, - 'password': None, - }, -} - - -config_filename = os.environ.get('QVARN_CONFIG', DEFAULT_CONFIG_FILE) -actual_config = read_config(config_filename) -config = dict(default_config) -config.update(actual_config or {}) -check_config(config) -qvarn.setup_logging(config) -qvarn.log.log('info', msg_text='Qvarn backend starting') - -subject = qvarn.ResourceType() -subject.from_spec({ - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'id': '', - 'type': '', - 'revision': '', - 'random_id': '', - 'names': [ - { - 'full_name': '', - 'sort_key': '', - 'titles': [''], - 'given_names': [''], - 'surnames': [''], - }, - ], - }, - 'subpaths': { - 'sub': { - 'prototype': { - 'subfield': '', - }, - }, - 'blob': { - 'prototype': { - 'body': 'blob', - 'content-type': '', - }, - }, - }, - 'files': [ - 'blob', - ], - }, - ], -}) - -resource_types = qvarn.load_resource_types(config['resource-type-dir']) - -if config['memory-database']: - store = qvarn.MemoryObjectStore() -else: - sql = qvarn.PostgresAdapter() - sql.connect(**config['database']) - store = qvarn.PostgresObjectStore(sql) - -api = qvarn.QvarnAPI() -api.set_base_url(config['baseurl']) -api.set_object_store(store) -api.add_resource_type(subject) -for rt in resource_types: - api.add_resource_type(rt) - -app = apifw.create_bottle_application( - api, counter, dict_logger, config, resource_types) - -# If we are running this program directly with Python, and not via -# gunicorn, we can use the Bottle built-in debug server, which can -# make some things easier to debug. - -if __name__ == '__main__': - print('running in debug mode') - app.run(host='127.0.0.1', port=12765) diff --git a/qvarn/collection.py b/qvarn/collection.py deleted file mode 100644 index 8a4f863..0000000 --- a/qvarn/collection.py +++ /dev/null @@ -1,300 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class CollectionAPI: - - object_keys = { - 'obj_id': str, - 'subpath': str, - } - - def __init__(self): - self._store = None - self._type = None - self._proto = None - self._idgen = qvarn.ResourceIdGenerator() - - def set_object_store(self, store): - self._store = store - store.create_store(**self.object_keys) - - def set_resource_type(self, rt): - assert isinstance(rt, qvarn.ResourceType) - self._type = rt - self._proto = self._type.get_latest_prototype() - - def get_type(self): - return self._type - - def get_type_name(self): - return self._type.get_type() - - def post(self, obj): - v = qvarn.Validator() - v.validate_new_resource(obj, self.get_type()) - - new_obj = self._new_object(self._proto, obj) - new_obj['id'] = self._invent_id(obj['type']) - new_obj['revision'] = self._invent_id('revision') - self._create_object(new_obj, obj_id=new_obj['id'], subpath='') - - rt = self.get_type() - subprotos = rt.get_subpaths() - for subpath, subproto in subprotos.items(): - empty = self._new_object(subproto, {}) - self._create_object(empty, obj_id=new_obj['id'], subpath=subpath) - - return new_obj - - def _create_object(self, obj, **keys): - assert set(keys.keys()) == set(self.object_keys.keys()) - qvarn.log.log( - 'debug', msg_text='Collection._create_object', obj=obj, keys=keys) - self._store.create_object(obj, **keys) - - def _new_object(self, proto, obj): - return qvarn.add_missing_fields(proto, obj) - - def _invent_id(self, resource_type): - return self._idgen.new_id(resource_type) - - def get(self, obj_id): - return self._get_object(obj_id, '') - - def get_subresource(self, obj_id, subpath): - return self._get_object(obj_id, subpath) - - def _get_object(self, obj_id, subpath): - keys = { - 'obj_id': obj_id, - 'subpath': subpath, - } - qvarn.log.log( - 'debug', msg_text='CollectionAPI._get_object', keys=keys) - objs = self._store.get_objects(**keys) - assert len(objs) <= 1 - if objs: - return objs[0] - raise NoSuchResource(**keys) - - def delete(self, obj_id): - self.get(obj_id) - self._store.remove_objects(obj_id=obj_id) - - def list(self): - oftype = qvarn.Equal('type', self.get_type_name()) - matches = self._store.find_objects(oftype) - qvarn.log.log('xxx', matches=matches, type=self.get_type_name()) - return { - 'resources': [ - {'id': obj['id']} - for _, obj in matches - if obj['type'] == self.get_type_name() - ] - } - - def put(self, obj): - v = qvarn.Validator() - v.validate_resource_update(obj, self.get_type()) - - old = self.get(obj['id']) - if old['revision'] != obj['revision']: - raise WrongRevision(obj['revision'], old['revision']) - - new_obj = dict(obj) - new_obj['revision'] = self._invent_id('revision') - self._store.remove_objects(obj_id=new_obj['id'], subpath='') - self._create_object(new_obj, obj_id=new_obj['id'], subpath='') - - return new_obj - - def put_subresource(self, sub_obj, subpath=None, **keys): - assert subpath is not None - obj_id = keys.pop('obj_id') - revision = keys.pop('revision') - parent = self.get(obj_id) - if parent['revision'] != revision: - raise WrongRevision(revision, parent['revision']) - keys = { - 'obj_id': obj_id, - 'subpath': subpath, - } - self._store.remove_objects(**keys) - self._create_object(sub_obj, **keys) - - parent = self._update_revision(obj_id) - new_sub = dict(sub_obj) - new_sub['revision'] = parent['revision'] - return new_sub - - def _update_revision(self, obj_id): - obj = self.get(obj_id) - obj['revision'] = self._invent_id('revision') - self._store.remove_objects(obj_id=obj_id, subpath='') - self._create_object(obj, obj_id=obj_id, subpath='') - return obj - - def search(self, search_criteria): - if not search_criteria: - raise NoSearchCriteria() - - p = qvarn.SearchParser() - sp = p.parse(search_criteria) - if sp.cond is None: - sp.cond = qvarn.Equal('type', self.get_type_name()) - - def pick_all(obj): - return obj - - def pick_id(obj): - return { - 'id': obj['id'], - } - - def pick_some_from_object(obj, fields): - return { - key: obj[key] - for key in obj - if key in fields - } - - def pick_some(fields): - return lambda obj: pick_some_from_object(obj, fields) - - if sp.show_all: - pick_fields = pick_all - elif sp.show_fields: - show_what = sp.show_fields + ['id'] - pick_fields = pick_some(show_what) - else: - pick_fields = pick_id - - # FIXME: This is needed because Qvarn API stupidly requires - # all fields to actually be defined by the resource type. If - # we drop that, we can drop this check, but that needs to be a - # managed transition, and for now we can't just drop it. - self._check_fields_are_allowed(sp.cond) - - unsorted = self._find_matches(sp.cond) - if sp.sort_keys: - result = self._sort_objects(unsorted, sp.sort_keys) - else: - result = unsorted - - if sp.offset is None and sp.limit is None: - chosen = result - elif sp.offset is None and sp.limit is not None: - chosen = result[:sp.limit] - elif sp.offset is not None and sp.limit is None: - chosen = result[sp.offset:] - elif sp.offset is not None and sp.limit is not None: - chosen = result[sp.offset:sp.offset+sp.limit] - - picked = [pick_fields(o) for o in chosen] - - qvarn.log.log( - 'trace', msg_text='Collection.search, sorted', - result=picked) - - return picked - - def _check_fields_are_allowed(self, cond): - names = set(self._get_names_from_cond(cond)) - allowed = set(self._get_allowed_names()) - for name in names.difference(allowed): - raise UnknownSearchField(name) - - def _get_names_from_cond(self, cond): - for c in qvarn.flatten(cond): - name = getattr(c, 'name') - if name is not None: - yield name - - def _get_allowed_names(self): - rt = self.get_type() - proto = rt.get_latest_prototype() - for name in self._get_names_from_prototype(proto): - yield name - - for subpath in rt.get_subpaths(): - subproto = rt.get_subprototype(subpath) - for name in self._get_names_from_prototype(subproto): - yield name - - def _get_names_from_prototype(self, proto): - schema = qvarn.schema(proto) - for t in schema: - for name in t[0]: - yield name - - def _find_matches(self, cond): - matches = self._store.find_objects(cond) - qvarn.log.log('xxx', matches=matches) - obj_ids = self._uniq(keys['obj_id'] for keys, _ in matches) - objects = [ - self._get_object(obj_id=obj_id, subpath='') - for obj_id in obj_ids - ] - return [o for o in objects if o['type'] == self.get_type_name()] - - def _uniq(self, items): - seen = set() - for item in items: - if item not in seen: - yield item - seen.add(item) - - def _sort_objects(self, objects, sort_keys): - def object_sort_key(obj, fields): - return [ - (key, value) - for key, value in qvarn.flatten_object(obj) - if key in fields - ] - - return sorted(objects, key=lambda o: object_sort_key(o, sort_keys)) - - -class WrongRevision(Exception): - - def __init__(self, actual, expected): - super().__init__( - 'PUTted objects must have correct revision set: ' - 'got {}, expected {}'.format(actual, expected)) - - -class NoSuchResource(Exception): - - def __init__(self, **keys): - keys_str = ', '.join( - '{}={!r}'.format(key, keys[key]) for key in sorted(keys)) - super().__init__("There is no resource with keys {}".format(keys_str)) - - -class UnknownSearchField(Exception): - - def __init__(self, field): - super().__init__('There is no field {}'.format(field)) - self.field = field - - -class NoSearchCriteria(Exception): - - def __init__(self): - super().__init__('No search criteria was given') diff --git a/qvarn/collection_tests.py b/qvarn/collection_tests.py deleted file mode 100644 index abc776e..0000000 --- a/qvarn/collection_tests.py +++ /dev/null @@ -1,448 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import itertools -import unittest - -import qvarn - - -class CollectionAPITests(unittest.TestCase): - - def setUp(self): - self.store = qvarn.MemoryObjectStore() - - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'type': '', - 'id': '', - 'revision': '', - 'full_name': '', - 'names': [ - { - 'sort_key': '', - }, - ], - 'things': [ - { - 'things': '', - 'other': '', - }, - ], - }, - 'subpaths': { - 'sub': { - 'prototype': { - 'subfield': '', - }, - }, - }, - }, - ], - } - self.rt = qvarn.ResourceType() - self.rt.from_spec(spec) - - self.coll = qvarn.CollectionAPI() - self.coll.set_object_store(self.store) - self.coll.set_resource_type(self.rt) - - def test_returns_specified_type(self): - self.assertEqual(self.coll.get_type(), self.rt) - - def test_returns_specified_type_name(self): - self.assertEqual(self.coll.get_type_name(), self.rt.get_type()) - - def test_post_raises_error_if_type_not_given(self): - obj = { - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.NoType): - self.coll.post(obj) - - def test_post_raises_error_if_type_is_not_expected_one(self): - obj = { - 'type': 'unperson', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.WrongType): - self.coll.post(obj) - - def test_post_raises_error_if_id_given(self): - obj = { - 'id': 'object-1', - 'type': 'subject', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.HasId): - self.coll.post(obj) - - def test_post_raises_error_if_revision_given(self): - obj = { - 'revision': 'rev-1', - 'type': 'subject', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.HasRevision): - self.coll.post(obj) - - def test_post_creates_a_new_resource(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - self.assertTrue(new_obj['id']) - self.assertTrue(new_obj['revision']) - self.assertEqual(new_obj['things'], []) - self.assertEqual(new_obj, self.coll.get(new_obj['id'])) - - sub = self.coll.get_subresource(new_obj['id'], 'sub') - self.assertEqual(sub, {'subfield': ''}) - - def test_post_creates_a_new_resource_with_dict_list(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - 'things': [ - { - 'other': 'foo', - }, - ], - } - new_obj = self.coll.post(obj) - self.assertTrue(new_obj['id']) - self.assertTrue(new_obj['revision']) - self.assertEqual( - new_obj['things'], - [ - { - 'things': '', - 'other': 'foo', - }, - ] - ) - self.assertEqual(new_obj, self.coll.get(new_obj['id'])) - - def test_post_creates_a_new_id_revision_every_time(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj1 = self.coll.post(obj) - new_obj2 = self.coll.post(obj) - self.assertNotEqual(new_obj1, new_obj2) - - def test_get_raise_error_if_not_found(self): - with self.assertRaises(qvarn.NoSuchResource): - self.coll.get('no-such-object-id') - - def test_deleting_nonexisent_resource_raise_error(self): - with self.assertRaises(qvarn.NoSuchResource): - self.coll.delete('no-such-object-id') - - def test_deleting_resource_makes_it_go_away(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - self.coll.delete(new_obj['id']) - with self.assertRaises(qvarn.NoSuchResource): - self.coll.get(new_obj['id']) - - def test_listing_objects_returns_empty_list_initially(self): - self.assertEqual(self.coll.list(), {'resources': []}) - - def test_listing_objects_returns_list_of_existing_object(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - - # Add an object of a different type so that we can make sure - # only objects of the right type are returned. - wrong_type = { - 'type': 'unperson', - 'full_name': 'James Bond', - } - self.store.create_object(wrong_type, obj_id='007') - - self.assertEqual( - self.coll.list(), - { - 'resources': [ - { - 'id': new_obj['id'] - } - ] - } - ) - - def test_putting_resource_without_id_raises_error(self): - obj = { - 'revision': 'revision-1', - 'type': 'subject', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.NoId): - self.coll.put(obj) - - def test_putting_resource_without_type_raises_error(self): - obj = { - 'id': 'object-id-1', - 'revision': 'revision-1', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.NoType): - self.coll.put(obj) - - def test_putting_resource_with_wrong_type_raises_error(self): - obj = { - 'id': 'object-id-1', - 'type': 'notasubject', - 'revision': 'revision-1', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.WrongType): - self.coll.put(obj) - - def test_putting_resource_without_revision_raises_error(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - del new_obj['revision'] - with self.assertRaises(qvarn.NoRevision): - self.coll.put(new_obj) - - def test_putting_resource_with_wrong_revision_raises_error(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - obj2 = dict(new_obj) - obj2['revision'] = 'this-revision-never-happens-randomly' - with self.assertRaises(qvarn.WrongRevision): - self.coll.put(obj2) - - def test_putting_nonexistent_resource_raises_error(self): - obj = { - 'id': 'object-id-1', - 'revision': 'revision-1', - 'type': 'subject', - 'full_name': 'James Bond', - } - with self.assertRaises(qvarn.NoSuchResource): - self.coll.put(obj) - - def test_putting_resource_updates_resource(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - obj2 = dict(new_obj) - obj2['full_name'] = 'Alfred Newman' - newer_obj = self.coll.put(obj2) - self.assertNotEqual(new_obj['revision'], newer_obj['revision']) - self.assertEqual(self.revisionless(obj2), self.revisionless(newer_obj)) - - def revisionless(self, obj): - return { - key: value - for key, value in obj.items() - if key != 'revision' - } - - def test_putting_subresource(self): - parent = { - 'type': 'subject', - 'full_name': 'James Bond', - } - sub = { - 'subfield': 'subvalue', - } - new_obj = self.coll.post(parent) - obj_id = new_obj['id'] - revision = new_obj['revision'] - new_sub = self.coll.put_subresource( - sub, subpath='sub', obj_id=obj_id, revision=revision) - self.assertNotEqual(new_sub['revision'], revision) - self.assertEqual(self.coll.get_subresource(obj_id, 'sub'), sub) - - new_parent = self.coll.get(obj_id) - self.assertNotEqual(new_parent['revision'], revision) - self.assertEqual(new_parent['revision'], new_sub['revision']) - - def test_putting_subresource_raises_error_without_parent_object(self): - sub = { - 'subfield': 'subvalue', - } - with self.assertRaises(qvarn.NoSuchResource): - self.coll.put_subresource( - sub, obj_id='unknown', revision='unknown', subpath='sub') - - def test_putting_subresource_raises_error_if_revision_is_wrong(self): - parent = { - 'type': 'subject', - 'full_name': 'James Bond', - } - sub = { - 'subfield': 'subvalue', - } - new_obj = self.coll.post(parent) - with self.assertRaises(qvarn.WrongRevision): - self.coll.put_subresource( - sub, obj_id=new_obj['id'], revision='wrong', subpath='sub') - - def test_search_with_empty_criteria_raises_error(self): - with self.assertRaises(qvarn.NoSearchCriteria): - self.coll.search('') - - def test_search_with_comparison_on_unknown_field_raises_error(self): - with self.assertRaises(qvarn.UnknownSearchField): - self.coll.search('exact/DOESNOTEXIST/sanity') - - def test_search_without_matches_returns_empty_list(self): - self.assertEqual(self.coll.search('exact/full_name/nomatch'), []) - - def test_search_return_matching_resource_ids(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - matches = self.coll.search('exact/full_name/James Bond') - self.assertEqual(matches, [{'id': new_obj['id']}]) - - def test_search_ignores_case(self): - obj = { - 'type': 'subject', - 'full_name': 'JAMES', - } - new_obj = self.coll.post(obj) - matches = self.coll.search('exact/full_name/james') - self.assertEqual(matches, [{'id': new_obj['id']}]) - - def test_search_return_matching_resources_themselves(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - wanted = { - 'id': new_obj['id'], - 'full_name': new_obj['full_name'], - } - matches = self.coll.search('exact/full_name/James Bond/show/full_name') - self.assertEqual(matches, [wanted]) - - def test_search_return_matching_resources_if_subresource_matches(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - sub = { - 'subfield': 'xyzzy', - } - - new_obj = self.coll.post(obj) - self.coll.put_subresource( - sub, subpath='sub', obj_id=new_obj['id'], - revision=new_obj['revision']) - matches = self.coll.search('exact/full_name/James Bond/show_all') - self.assertEqual(matches, [new_obj]) - - def test_search_return_full_resources(self): - obj = { - 'type': 'subject', - 'full_name': 'James Bond', - } - new_obj = self.coll.post(obj) - matches = self.coll.search('exact/full_name/James Bond/show_all') - self.assertEqual(matches, [new_obj]) - - def test_search_sorts(self): - names = ['Alfred Pennyweather', 'James Bond', 'Jason Bourne'] - for perm in itertools.permutations(names): - self.setUp() - for name in perm: - obj = { - 'type': 'subject', - 'full_name': 'Spy', - 'names': [ - { - 'sort_key': name, - }, - ], - } - self.coll.post(obj) - - matches = self.coll.search( - 'exact/full_name/Spy/show_all/sort/sort_key') - self.assertEqual(len(matches), 3) - self.assertEqual( - [m['names'][0]['sort_key'] for m in matches], - names) - - def test_limit_without_sort(self): - with self.assertRaises(qvarn.SearchParserError): - self.coll.search('/limit/1') - - def test_offset_without_sort(self): - with self.assertRaises(qvarn.SearchParserError): - self.coll.search('/offset/1') - - def test_search_with_limit_only(self): - objs = self.create_objects(['1', '2', '3']) - search = 'sort/full_name/show_all' - matches = self.coll.search(search + '/limit/2') - self.assertEqual(len(matches), 2) - self.assertEqual(matches, objs[:2]) - - def test_search_with_offset_only(self): - objs = self.create_objects(['1', '2', '3']) - search = 'sort/full_name/show_all' - matches = self.coll.search(search + '/offset/2') - self.assertEqual(len(matches), 1) - self.assertEqual(matches, objs[2:]) - - def test_search_with_offset_and_limit(self): - objs = self.create_objects(['1', '2', '3']) - search = 'sort/full_name/show_all' - matches = self.coll.search(search + '/offset/1/limit/1') - self.assertEqual(len(matches), 1) - self.assertEqual(matches, objs[1:2]) - - def create_objects(self, names): - objs = [] - for name in names: - obj = { - 'type': 'subject', - 'full_name': name, - } - new_obj = self.coll.post(obj) - objs.append(new_obj) - return objs diff --git a/qvarn/file_router.py b/qvarn/file_router.py deleted file mode 100644 index b7e4507..0000000 --- a/qvarn/file_router.py +++ /dev/null @@ -1,98 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class FileRouter(qvarn.Router): - - def __init__(self): - super().__init__() - self._store = None - self._parent_coll = None - self._subpath = None - - def set_subpath(self, subpath): - self._subpath = subpath - - def set_parent_collection(self, parent_coll): - self._parent_coll = parent_coll - - def set_object_store(self, store): - self._store = store - - def get_routes(self): - rt = self._parent_coll.get_type() - file_path = '{}/<id>/{}'.format(rt.get_path(), self._subpath) - return [ - { - 'method': 'GET', - 'path': file_path, - 'callback': self._get_file, - }, - { - 'method': 'PUT', - 'path': file_path, - 'callback': self._put_file, - }, - ] - - def _get_file(self, *args, **kwargs): - obj_id = kwargs['id'] - try: - obj = self._parent_coll.get(obj_id) - sub_obj = self._parent_coll.get_subresource(obj_id, self._subpath) - blob = self._store.get_blob(obj_id=obj_id, subpath=self._subpath) - except (qvarn.NoSuchResource, qvarn.NoSuchObject) as e: - return qvarn.no_such_resource_response(str(e)) - headers = { - 'Content-Type': sub_obj['content_type'], - 'Revision': obj['revision'], - } - return qvarn.ok_response(blob, headers) - - def _put_file(self, content_type, body, *args, **kwargs): - obj_id = kwargs['id'] - - # FIXME: add header getting to apifw - import bottle - revision = bottle.request.get_header('Revision') - - obj = self._parent_coll.get(obj_id) - if obj['revision'] != revision: - qvarn.log.log( - 'error', - msg_text='Client gave wrong revision', - revision_from_client=revision, - current_revision=obj['revision']) - return qvarn.conflict_response( - 'Bad revision {}'.format(revision)) - - sub_obj = self._parent_coll.get_subresource(obj_id, self._subpath) - sub_obj['content_type'] = content_type - new_sub = self._parent_coll.put_subresource( - sub_obj, subpath=self._subpath, obj_id=obj_id, revision=revision) - - try: - self._store.remove_blob(obj_id=obj_id, subpath=self._subpath) - self._store.create_blob(body, obj_id=obj_id, subpath=self._subpath) - except qvarn.NoSuchObject as e: - return qvarn.no_such_resource_response(str(e)) - - headers = { - 'Revision': new_sub['revision'], - } - return qvarn.ok_response('', headers) diff --git a/qvarn/idgen.py b/qvarn/idgen.py deleted file mode 100644 index f22332f..0000000 --- a/qvarn/idgen.py +++ /dev/null @@ -1,147 +0,0 @@ -# idgen.py - generate resource identifiers -# -# Copyright 2015, 2016 Suomen Tilaajavastuu Oy -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -# Implementation discussion -# ========================= -# -# A resource identifier must be unique, efficient, secure, and -# private. These not in priority order: all requirements must be -# fulfilled. Additionally, it would be good for identifiers to be -# practical. -# -# Uniqueness and efficiency -# ------------------------- -# -# Uniqueness can be provided by having a central database in which all -# identifiers exist, and adding new ones there. This is a performance -# bottleneck. We choose to rely on good, large random numbers instead. -# Quoting Wikipedia on random UUIDs: -# -# To put these numbers into perspective, the annual risk of a -# given person being hit by a meteorite is estimated to be one -# chance in 17 billion,[4] which means the probability is about -# 0.00000000006 (6 * 10**11), equivalent to the odds of creating -# a few tens of trillions of UUIDs in a year and having one -# duplicate. In other words, only after generating 1 billion -# UUIDs every second for the next 100 years, the probability of -# creating just one duplicate would be about 50%. -# -# See http://en.wikipedia.org/wiki/Universally_unique_identifier. -# -# We do not used the UUID format specifically, but the principle -# holds. If get random numbers from a high-quality source, and use at -# least 128 bits of randomness (UUID4 uses 122), we're good. A bug in -# other parts of our software, the database software, the operating -# system, hardware, or operational procedures is more likely to cause -# duplicates. -# -# Security and privacy -# -------------------- -# -# We want our identifiers to avoid leaking information. A linear -# counter, for example, would leak information: the number of resource -# objects created. We also can't use external identifiers, such as -# social security identifiers, as parts of resource identifiers, as -# this would leak actual sensitive information. Using random numbers -# is perfect. -# -# For added privacy, it would be good if different API clients would -# get different identifiers for the same resource. This would make it -# more difficult for them to combine data and endanger people's -# privacy. This level of protection is currently not handled at all, -# and in any case will probably not handled by this class. Instead, if -# we want to do this, we'll add a translation layer that changes -# internal identifiers to be per-client ones, and back again. -# -# Error checking -# -------------- -# -# Inevitably, resource identifiers will show up in URLs, in log files, -# and be communicated by humans using writing or voice. To make these -# things easier, it would be good to be able to verify that an -# identifier looks correct, by adding error checking into the identifier. -# -# Further, it would be good to have type information: is this -# identifier one for a person or an organisation? This can catch -# attempts at using an identifier for the wrong type of resource. -# -# Identifier strucure -# ------------------- -# -# Based on the above discussion, we define the following structure for -# a resource identifier: -# -# * 16 bits of type field -# * 128 bits of randomness -# * 32 bits of error checking -# -# The type field is the top 16 bits of a SHA-512 of the resources type: -# effectively this: -# -# hashlib.sha512('person').hexdigest()[:4] -# -# The random bits are read directly from /dev/urandom. Python provides -# os.urandom and uuid.uuid4, which could either be used, but to avoid -# having to trust Python's implementation, we read /dev/urandom -# directly. -# -# The error checking is done by computing the SHA-512 of the rest of -# the identifier and taking the top 32 bits: -# -# hashlib.sha512(rest).hexdigest()[:8] -# -# For human convenience, we allow identifiers (which are effectively -# very large hexadecimal numbers) to be represented in upper or lower -# case, and that any non-hexdigits are ignored. -# -# The canonical form of an identifier (in this case, for a person): -# -# 0035-94c4f55599453307002f0731e0b67999-9ffa4cf4 - - -import hashlib -import os - - -class ResourceIdGenerator: - - '''Generate resource identifiers and revisions.''' - - def new_id(self, resource_type): - '''Generate a new identifier.''' - - type_field = self._encode_type(resource_type) - random_field = self._get_randomness() - checksum_field = self._compute_checksum(type_field + random_field) - return self._canonical_form(type_field, random_field, checksum_field) - - def _encode_type(self, resource_type): - return hashlib.sha512(resource_type.encode('ascii')).hexdigest()[:4] - - def _get_randomness(self): - num_bits = 128 - num_bytes = num_bits // 8 - random_bytes = os.urandom(num_bytes) - return random_bytes.hex() - - def _compute_checksum(self, rest): - return hashlib.sha512(rest.encode('ascii')).hexdigest()[:8] - - def _canonical_form(self, type_field, random_field, checksum_field): - return '{0}-{1}-{2}'.format(type_field, random_field, checksum_field) diff --git a/qvarn/idgen_tests.py b/qvarn/idgen_tests.py deleted file mode 100644 index 944db1c..0000000 --- a/qvarn/idgen_tests.py +++ /dev/null @@ -1,36 +0,0 @@ -# idgen_tests.py - unit tests for ResourceIdGenerator -# -# Copyright 2015, 2016 Suomen Tilaajavastuu Oy -# Copyright 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - -import qvarn - - -class ResourceIdGeneratorTests(unittest.TestCase): - - def test_returns_a_unicode_string(self): - rig = qvarn.ResourceIdGenerator() - resource_id = rig.new_id('person') - self.assertTrue(isinstance(resource_id, str)) - - def test_returns_new_values_each_time(self): - rig = qvarn.ResourceIdGenerator() - id_1 = rig.new_id('person') - id_2 = rig.new_id('person') - self.assertNotEqual(id_1, id_2) diff --git a/qvarn/notification_router.py b/qvarn/notification_router.py deleted file mode 100644 index b96e38e..0000000 --- a/qvarn/notification_router.py +++ /dev/null @@ -1,254 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class NotificationRouter(qvarn.Router): - - def __init__(self): - super().__init__() - self._baseurl = None - self._store = None - self._parent_coll = None - self._listener_coll = None - - def set_baseurl(self, baseurl): - self._baseurl = baseurl - - def set_parent_collection(self, parent_coll): - self._parent_coll = parent_coll - - def set_object_store(self, store, listener_rt): - self._store = store - listeners = qvarn.CollectionAPI() - listeners.set_object_store(self._store) - listeners.set_resource_type(listener_rt) - self._listener_coll = listeners - - def get_routes(self): - rt = self._parent_coll.get_type() - listeners_path = '{}/listeners'.format(rt.get_path()) - listener_id_path = '{}/<listener_id>'.format(listeners_path) - notifications_path = '{}/notifications'.format(listener_id_path) - notification_id_path = '{}/<notification_id>'.format( - notifications_path) - - return [ - { - 'method': 'POST', - 'path': listeners_path, - 'callback': self._create_listener, - }, - { - 'method': 'GET', - 'path': listeners_path, - 'callback': self._get_listener_list, - }, - { - 'method': 'GET', - 'path': listener_id_path, - 'callback': self._get_a_listener, - }, - { - 'method': 'PUT', - 'path': listener_id_path, - 'callback': self._update_listener, - }, - { - 'method': 'DELETE', - 'path': listener_id_path, - 'callback': self._delete_listener, - }, - { - 'method': 'GET', - 'path': notifications_path, - 'callback': self._get_notifications_list, - }, - { - 'method': 'GET', - 'path': notification_id_path, - 'callback': self._get_a_notification, - }, - { - 'method': 'DELETE', - 'path': notification_id_path, - 'callback': self._delete_notification, - }, - ] - - def _create_listener(self, content_type, body, *args, **kwargs): - if content_type != 'application/json': - raise qvarn.NotJson(content_type) - - rt = self._listener_coll.get_type() - validator = qvarn.Validator() - try: - validator.validate_against_prototype( - rt.get_type(), body, rt.get_latest_prototype()) - except qvarn.ValidationError as e: - qvarn.log.log('error', msg_text=str(e), body=body) - return qvarn.bad_request_response(str(e)) - - if 'type' not in body: - body['type'] = 'listener' - - rtype = self._parent_coll.get_type_name() - if body.get('listen_on_type', rtype) != rtype: - return qvarn.bad_request_response( - 'listen_on_type does not have value {}'.format(rtype)) - body['listen_on_type'] = rtype - - result_body = self._listener_coll.post(body) - location = self._get_new_resource_location(result_body) - qvarn.log.log( - 'debug', msg_text='POST a new listener, result', - body=result_body, location=location) - return qvarn.created_response(result_body, location) - - def _get_new_resource_location(self, resource): - return '{}{}/listeners/{}'.format( - self._baseurl, self._parent_coll.get_type().get_path(), - resource['id']) - - def _get_listener_list(self, content_type, body, *args, **kwargs): - rtype = self._parent_coll.get_type_name() - listener_list = self._listener_coll.list()['resources'] - listener_ids = [listener['id'] for listener in listener_list] - listeners = [self._listener_coll.get(lid) for lid in listener_ids] - qvarn.log.log('trace', msg_text='xxx', listeners=listeners) - correct_ids = [ - {"id": listener['id']} - for listener in listeners - if listener['listen_on_type'] == rtype - ] - body = { - 'resources': correct_ids, - } - return qvarn.ok_response(body) - - def _get_a_listener(self, *args, **kwargs): - try: - obj = self._listener_coll.get(kwargs['listener_id']) - except qvarn.NoSuchResource as e: - return qvarn.no_such_resource_response(str(e)) - return qvarn.ok_response(obj) - - def _update_listener(self, content_type, body, *args, **kwargs): - if content_type != 'application/json': - raise qvarn.NotJson(content_type) - - if 'type' not in body: - body['type'] = 'listener' - - listener_id = kwargs['listener_id'] - if 'id' not in body: - body['id'] = listener_id - - validator = qvarn.Validator() - try: - validator.validate_resource_update( - body, self._listener_coll.get_type()) - except qvarn.ValidationError as e: - qvarn.log.log('error', msg_text=str(e), body=body) - return qvarn.bad_request_response(str(e)) - - try: - result_body = self._listener_coll.put(body) - except qvarn.WrongRevision as e: - return qvarn.conflict_response(str(e)) - except qvarn.NoSuchResource as e: - # We intentionally say bad request, instead of not found. - # This is to be compatible with old Qvarn. This may get - # changed later. - return qvarn.bad_request_response(str(e)) - - return qvarn.ok_response(result_body) - - def _delete_listener(self, *args, **kwargs): - listener_id = kwargs['listener_id'] - self._listener_coll.delete(listener_id) - for obj_id in self._find_notifications(listener_id): - self._store.remove_objects(obj_id=obj_id) - return qvarn.ok_response({}) - - def _find_notifications(self, listener_id): - cond = qvarn.All( - qvarn.Equal('type', 'notification'), - qvarn.Equal('listener_id', listener_id), - ) - obj_ids = [ - keys['obj_id'] - for keys, _ in self._store.find_objects(cond) - ] - qvarn.log.log( - 'trace', msg_text='Found notifications', - notifications=obj_ids) - return obj_ids - - def _get_notifications_list(self, *args, **kwargs): - def timestamp(pair): - _, obj = pair - return obj['timestamp'] - - listener_id = kwargs['listener_id'] - cond = qvarn.All( - qvarn.Equal('type', 'notification'), - qvarn.Equal('listener_id', listener_id) - ) - pairs = self._store.find_objects(cond) - ordered = sorted(pairs, key=timestamp) - body = { - 'resources': [ - { - 'id': keys['obj_id'] - } - for keys, _ in ordered - ] - } - return qvarn.ok_response(body) - - def _get_a_notification(self, *args, **kwargs): - listener_id = kwargs['listener_id'] - notification_id = kwargs['notification_id'] - cond = qvarn.All( - qvarn.Equal('type', 'notification'), - qvarn.Equal('listener_id', listener_id), - qvarn.Equal('id', notification_id), - ) - pairs = self._store.find_objects(cond) - if len(pairs) == 0: - return qvarn.no_such_resource_response(notification_id) - if len(pairs) > 1: - raise qvarn.TooManyResources(notification_id) - return qvarn.ok_response(pairs[0][1]) - - def _delete_notification(self, *args, **kwargs): - listener_id = kwargs['listener_id'] - notification_id = kwargs['notification_id'] - cond = qvarn.All( - qvarn.Equal('type', 'notification'), - qvarn.Equal('listener_id', listener_id), - qvarn.Equal('id', notification_id), - ) - for keys, _ in self._store.find_objects(cond): - values = { - key: keys[key] - for key in keys - if isinstance(keys[key], str) - } - self._store.remove_objects(**values) - return qvarn.ok_response({}) diff --git a/qvarn/objstore.py b/qvarn/objstore.py deleted file mode 100644 index d88774e..0000000 --- a/qvarn/objstore.py +++ /dev/null @@ -1,413 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import json - - -import qvarn - - -class ObjectStoreInterface: # pragma: no cover - - '''Store and retrieve JSON-like objects. - - A JSON-like object is a Python dict whose keys are strings, and - values corresponding to they keys can serialised into JSON, so - they're strings, integers, booleans, or JSON-like objects, or - lists of such values. JSON would support more types, but Qvarn - doesn't need them. Value strings (but not key names) may be - Unicode text or binary strings. - - The object store stores the JSON-like object, and a set of keys - that identify the object. The caller gets to define the keys. The - keys must be strings. There can be any number of keys, but there - must be at least one. The caller gets to define the keys and their - meaning. The allowed keys (and their types) are set when the store - is created, using the create_store method. - - Objects may be retrieved or removed using any subset of keys. All - maching objects are retrieved or removed. - - Objects mey be found using conditions, implemented by subclasses - of the qvarn.Condition class. Various condidions may be combined - arbitrarily. - - This class is for defining the ObjectStore interface. There is an - in-memory variant, for use in unit tests, and a version using - PostgreSQL for production use. - - ''' - - def create_store(self, **keys): - raise NotImplementedError() - - def check_keys_have_str_type(self, **keys): - for key in keys: - if keys[key] != str: - raise WrongKeyType(key, keys[key]) - - def get_known_keys(self): - raise NotImplementedError() - - def check_all_keys_are_allowed(self, **keys): - known_keys = self.get_known_keys() - for key in keys: - if key not in known_keys: - raise UnknownKey(key) - - def check_value_types(self, **keys): - known_keys = self.get_known_keys() - for key in keys: - if type(keys[key]) is not known_keys[key]: - raise KeyValueError(key, keys[key]) - - def create_object(self, obj, auxtable=True, **keys): - raise NotImplementedError() - - def remove_objects(self, **keys): - raise NotImplementedError() - - def get_objects(self, **keys): - raise NotImplementedError() - - def find_objects(self, cond): - raise NotImplementedError() - - def create_blob(self, blob, subpath=None, **keys): - raise NotImplementedError() - - def get_blob(self, subpath=None, **keys): - raise NotImplementedError() - - def remove_blob(self, blob, subpath=None, **keys): - raise NotImplementedError() - - -class MemoryObjectStore(ObjectStoreInterface): - - def __init__(self): - self._objs = [] - self._blobs = [] - self._known_keys = {} - - def get_known_keys(self): - return self._known_keys - - def create_store(self, **keys): - self.check_keys_have_str_type(**keys) - qvarn.log.log( - 'trace', msg_text='Creating store', keys=repr(keys), exc_info=True) - self._known_keys = keys - - def create_object(self, obj, auxtable=True, **keys): - qvarn.log.log( - 'trace', msg_text='Creating object', object=repr(obj), keys=keys) - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - self._check_unique_object(**keys) - self._objs.append((obj, keys)) - - def _check_unique_object(self, **keys): - for _, k in self._objs: - if self._keys_match(k, keys): - raise KeyCollision(k) - - def create_blob(self, blob, **keys): - qvarn.log.log('trace', msg_text='Creating blob', keys=keys) - subpath = keys.pop('subpath') - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - self._check_unique_blob(subpath, **keys) - if not self.get_objects(**keys): - raise NoSuchObject(keys) - self._blobs.append((blob, subpath, keys)) - - def _check_unique_blob(self, subpath, **keys): - for _, s, k in self._blobs: - if self._keys_match(k, keys) and s == subpath: - raise BlobKeyCollision(subpath, k) - - def get_blob(self, **keys): - subpath = keys.pop('subpath') - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - blobs = [ - b - for b, s, k in self._blobs - if self._keys_match(k, keys) and s == subpath - ] - assert len(blobs) <= 1 - if not blobs: - raise NoSuchObject(keys) - return blobs[0] - - def remove_blob(self, **keys): - subpath = keys.pop('subpath') - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - self._blobs = [ - b - for b, s, k in self._blobs - if not self._keys_match(k, keys) or s != subpath - ] - - def remove_objects(self, **keys): - self.check_all_keys_are_allowed(**keys) - self._objs = [ - (o, k) for o, k in self._objs if not self._keys_match(k, keys)] - - def get_objects(self, **keys): - self.check_all_keys_are_allowed(**keys) - return [o for o, k in self._objs if self._keys_match(k, keys)] - - def _keys_match(self, got_keys, wanted_keys): - for key in wanted_keys.keys(): - if got_keys.get(key) != wanted_keys[key]: - return False - return True - - def find_objects(self, cond): - return [(keys, obj) for obj, keys in self._objs if cond.matches(obj)] - - -class PostgresObjectStore(ObjectStoreInterface): # pragma: no cover - - _table = '_objects' - _auxtable = '_aux' - _blobtable = '_blobs' - - def __init__(self, sql): - self._sql = sql - self._keys = None - - def get_known_keys(self): - return self._keys - - def create_store(self, **keys): - self.check_keys_have_str_type(**keys) - self._keys = dict(keys) - qvarn.log.log( - 'info', msg_text='PostgresObjectStore.create_store', - keys=repr(keys)) - - # Create main table for objects. - self._create_table(self._table, self._keys, '_obj', dict) - - # Create helper table for fields at all depths. Needed by searches. - self._create_table(self._auxtable, self._keys, '_field', dict) - - # Create helper table for blobs. - self._create_table(self._blobtable, self._keys, '_blob', bytes) - - def _create_table(self, name, col_dict, col_name, col_type): - columns = dict(col_dict) - columns[col_name] = col_type - with self._sql.transaction() as t: - query = t.create_table(name, **columns) - t.execute(query, {}) - - def create_object(self, obj, auxtable=True, **keys): - qvarn.log.log( - 'info', msg_text='PostgresObjectStore.create_object', - obj=obj, keys=keys) - with self._sql.transaction() as t: - self._remove_objects_in_transaction(t, **keys) - self._insert_into_object_table(t, self._table, obj, **keys) - if auxtable: - self._insert_into_helper(t, self._auxtable, obj, **keys) - - def _insert_into_object_table(self, t, table_name, obj, **keys): - keys['_obj'] = json.dumps(obj) - column_names = list(keys.keys()) - query = t.insert_object(table_name, *column_names) - t.execute(query, keys) - - def _insert_into_helper(self, t, table_name, obj, **keys): - for field, value in flatten_object(obj): - x = { - 'name': field, - 'value': value, - } - keys['_field'] = json.dumps(x) - column_names = list(keys.keys()) - query = t.insert_object(table_name, *column_names) - t.execute(query, keys) - - def remove_objects(self, **keys): - qvarn.log.log( - 'info', msg_text='PostgresObjectStore.remove_objects', - keys=keys) - with self._sql.transaction() as t: - query = t.remove_objects(self._table, *keys.keys()) - t.execute(query, keys) - - query = t.remove_objects(self._auxtable, *keys.keys()) - t.execute(query, keys) - - def _remove_objects_in_transaction(self, t, **keys): - qvarn.log.log( - 'info', - msg_text='PostgresObjectStore._remove_objects_in_transaction', - keys=keys) - query = t.remove_objects(self._table, *keys.keys()) - t.execute(query, keys) - query = t.remove_objects(self._auxtable, *keys.keys()) - t.execute(query, keys) - - def get_objects(self, **keys): - qvarn.log.log( - 'info', msg_text='PostgresObjectStore.get_objects', - keys=keys) - with self._sql.transaction() as t: - return self._get_objects_in_transaction(t, **keys) - - def _get_objects_in_transaction(self, t, **keys): - query = t.select_objects(self._table, '_obj', *keys.keys()) - qvarn.log.log( - 'debug', msg_text='PostgresObjectStore.get_objects', - query=query, keys=keys) - cursor = t.execute(query, keys) - return [row['_obj'] for row in t.get_rows(cursor)] - - def find_objects(self, cond): - qvarn.log.log( - 'info', msg_text='PostgresObjectStore.find_objects', - cond=repr(cond)) - with self._sql.transaction() as t: - rows = self._find_helper(t, cond) - return [ - self._split_row(row) - for row in rows - if row['subpath'] == '' - ] - - def _find_helper(self, t, cond): - keys_columns = [key for key in self._keys if key != '_obj'] - query, values = t.select_objects_on_cond( - self._auxtable, cond, *keys_columns) - cursor = t.execute(query, values) - return t.get_rows(cursor) - - def _split_row(self, row): - keys = dict(row) - obj = row.pop('_obj') - return keys, obj - - def create_blob(self, blob, **keys): - qvarn.log.log('trace', msg_text='Creating blob', keys=keys) - - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - if not self.get_objects(**keys): - raise NoSuchObject(keys) - - with self._sql.transaction() as t: - column_names = list(keys.keys()) + ['_blob'] - query = t.insert_object(self._blobtable, *column_names) - - values = dict(keys) - values['_blob'] = blob - - t.execute(query, values) - - def get_blob(self, **keys): - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - - column_names = list(keys.keys()) - - with self._sql.transaction() as t: - query = t.select_objects(self._blobtable, '_blob', *column_names) - blobs = [bytes(row['_blob']) for row in t.execute(query, keys)] - if len(blobs) == 0: - raise NoSuchObject(keys) - return blobs - - def remove_blob(self, **keys): - self.check_all_keys_are_allowed(**keys) - self.check_value_types(**keys) - - column_names = list(keys.keys()) - with self._sql.transaction() as t: - query = t.remove_objects(self._blobtable, *column_names) - t.execute(query, keys) - - -class KeyCollision(Exception): - - def __init__(self, keys): - super().__init__('Cannot add object with same keys: %r' % keys) - - -class BlobKeyCollision(Exception): - - def __init__(self, subpath, keys): - super().__init__( - 'Cannot add blob with same keys: subpath=%s %r' % (subpath, keys)) - - -class UnknownKey(Exception): - - def __init__(self, key): - super().__init__('ObjectStore is not prepared for key %r' % key) - - -class WrongKeyType(Exception): - - def __init__(self, key, key_type): - super().__init__( - 'ObjectStore is not prepared for key %r of type %r, must be str' % - (key, key_type)) - - -class KeyValueError(Exception): - - def __init__(self, key, value): - super().__init__('Key %r value %r has the wrong type' % (key, value)) - - -class NoSuchObject(Exception): - - def __init__(self, keys): - super().__init__('No object/blob with keys {}'.format(keys)) - - -def flatten_object(obj): - # We sort only by the name, not the object in each pair in the - # list. Otherwise, if there are two fields with the same name but - # incompatible value types this will break. However, to guarantee - # that objects always result in the same flattened representation, - # we also compare the second field. For this, we convert the - # second value to a string with repr. This allows the second - # fields to be compared regardless of type. - - pairs = _flatten(obj) - unique_pairs = set(pairs) - sorted_pairs = sorted(unique_pairs, key=repr) - return list(sorted_pairs) - - -def _flatten(obj, obj_key=None): - if isinstance(obj, dict): - for key, value in obj.items(): - for x in _flatten(value, obj_key=key): - yield x - elif isinstance(obj, list): - for item in obj: - for x in _flatten(item, obj_key=obj_key): - yield x - else: - yield obj_key, obj diff --git a/qvarn/objstore_tests.py b/qvarn/objstore_tests.py deleted file mode 100644 index f8d631a..0000000 --- a/qvarn/objstore_tests.py +++ /dev/null @@ -1,244 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - -import qvarn - - -class ObjectStoreTests(unittest.TestCase): - - def setUp(self): - self.obj1 = { - 'name': 'this is my object', - } - self.obj2 = { - 'name': 'this is my other object', - } - self.blob1 = 'my first blob' - self.blob2 = 'my other blob' - - def create_store(self, **keys): - store = qvarn.MemoryObjectStore() - store.create_store(**keys) - return store - - def get_all_objects(self, store): - return [obj for _, obj in store.find_objects(qvarn.Yes())] - - def sorted_dicts(self, dicts): - return sorted(dicts, key=lambda d: sorted(d.items())) - - def test_is_initially_empty(self): - store = self.create_store(key=str) - self.assertEqual(self.get_all_objects(store), []) - - def test_refurses_nonstr_keys(self): - with self.assertRaises(qvarn.WrongKeyType): - self.create_store(key=int) - - def test_adds_object(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - self.assertEqual(self.get_all_objects(store), [self.obj1]) - - def test_adds_object_with_binary_data(self): - store = self.create_store(key=str) - self.obj1['data'] = bytes(range(0, 256)) - store.create_object(self.obj1, key='1st') - self.assertEqual(self.get_all_objects(store), [self.obj1]) - - def test_raises_error_for_surprising_keys(self): - store = self.create_store(key=str) - with self.assertRaises(qvarn.UnknownKey): - store.create_object(self.obj1, surprise='1st') - - def test_raises_error_adding_object_with_existing_keys(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - with self.assertRaises(qvarn.KeyCollision): - store.create_object(self.obj1, key='1st') - - def test_raises_error_adding_object_with_keys_of_wrong_type(self): - store = self.create_store(key=str) - with self.assertRaises(qvarn.KeyValueError): - store.create_object(self.obj1, key=1) - - def test_adds_objects_with_two_keys_with_one_key_the_same(self): - store = self.create_store(key1=str, key2=str) - store.create_object(self.obj1, key1='same', key2='1st') - store.create_object(self.obj2, key1='same', key2='2nd') - self.assertEqual(self.get_all_objects(store), [self.obj1, self.obj2]) - - def test_removes_only_object(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.remove_objects(key='1st') - self.assertEqual(self.get_all_objects(store), []) - - def test_gets_objects(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_object(self.obj2, key='2nd') - self.assertEqual(store.get_objects(key='1st'), [self.obj1]) - self.assertEqual(store.get_objects(key='2nd'), [self.obj2]) - - def test_gets_objects_using_only_one_key(self): - store = self.create_store(key1=str, key2=str) - store.create_object(self.obj1, key1='1st', key2='foo') - store.create_object(self.obj2, key1='2nd', key2='foo') - self.assertEqual(store.get_objects(key1='1st'), [self.obj1]) - self.assertEqual(store.get_objects(key1='2nd'), [self.obj2]) - self.assertEqual( - self.sorted_dicts(store.get_objects(key2='foo')), - self.sorted_dicts([self.obj1, self.obj2])) - - def test_removes_only_one_object(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_object(self.obj2, key='2nd') - store.remove_objects(key='1st') - self.assertEqual(self.get_all_objects(store), [self.obj2]) - - def test_finds_objects(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_object(self.obj2, key='2nd') - - cond = qvarn.Equal('name', self.obj1['name']) - objs = store.find_objects(cond) - self.assertEqual( - objs, - [({'key': '1st'}, self.obj1)] - ) - - def test_has_no_blob_initially(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - with self.assertRaises(qvarn.NoSuchObject): - store.get_blob(key='1st', subpath='blob') - - def test_add_blob_to_nonexistent_parent_fails(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - with self.assertRaises(qvarn.NoSuchObject): - store.create_blob(self.blob1, key='2nd', subpath='blob') - - def test_adds_blob(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_blob(self.blob1, key='1st', subpath='blob') - blob = store.get_blob(key='1st', subpath='blob') - self.assertEqual(blob, self.blob1) - - def test_add_blob_twice_fails(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_blob(self.blob1, key='1st', subpath='blob') - with self.assertRaises(qvarn.BlobKeyCollision): - store.create_blob(self.blob1, key='1st', subpath='blob') - - def test_removes_blob(self): - store = self.create_store(key=str) - store.create_object(self.obj1, key='1st') - store.create_blob(self.blob1, key='1st', subpath='blob') - store.remove_blob(key='1st', subpath='blob') - with self.assertRaises(qvarn.NoSuchObject): - store.get_blob(key='1st', subpath='blob') - - -class FlattenObjectsTests(unittest.TestCase): - - def test_flattens_simple_dict(self): - obj = { - 'foo': 'bar', - 'foobar': 42, - 'yo': True, - } - self.assertEqual( - qvarn.flatten_object(obj), - sorted([('foo', 'bar'), ('foobar', 42), ('yo', True)])) - - def test_flattens_deep_dict(self): - obj = { - 'foo': 'bar', - 'foos': [ - { - 'foo': 'bar2', - 'foos': [ - { - 'foo': 'bar3', - }, - ], - }, - ], - } - self.assertEqual( - qvarn.flatten_object(obj), - sorted([ - ('foo', 'bar'), - ('foo', 'bar2'), - ('foo', 'bar3'), - ])) - - -class FindObjectsTests(unittest.TestCase): - - def setUp(self): - pass - - def create_store(self, **keys): - store = qvarn.MemoryObjectStore() - store.create_store(**keys) - return store - - def test_finds_objects_matching_deeply_in_object(self): - obj1 = { - 'foo': 'foo-1', - 'bar': 'blah', - 'bars': [ - { - 'foo': 'bars.0', - 'bar': 'yo', - }, - { - 'foo': 'bars.1', - 'bar': 'bleurgh', - }, - ], - } - - obj2 = { - 'foo': 'foo-2', - 'bar': 'bother', - 'bars': [], - } - - keys1 = { - 'key': '1st', - } - - keys2 = { - 'key': '2nd', - } - - store = self.create_store(key=str) - store.create_object(obj1, **keys1) - store.create_object(obj2, **keys2) - - cond = qvarn.Equal('bar', 'yo') - objs = store.find_objects(cond) - self.assertEqual(objs, [(keys1, obj1)]) diff --git a/qvarn/resource_router.py b/qvarn/resource_router.py deleted file mode 100644 index c9df68a..0000000 --- a/qvarn/resource_router.py +++ /dev/null @@ -1,167 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class ResourceRouter(qvarn.Router): - - def __init__(self): - super().__init__() - self._coll = None - self._baseurl = None - self._notify = None - - def set_baseurl(self, baseurl): - self._baseurl = baseurl - - def set_collection(self, coll): - self._coll = coll - - def set_notifier(self, notify): - self._notify = notify - - def get_routes(self): - assert self._baseurl is not None - - rt = self._coll.get_type() - path = rt.get_path() - id_path = '{}/<id>'.format(path) - - return [ - { - 'method': 'POST', - 'path': path, - 'callback': self._create, - }, - { - 'method': 'PUT', - 'path': id_path, - 'callback': self._update, - }, - { - 'method': 'GET', - 'path': path, - 'callback': self._list, - }, - { - 'method': 'GET', - 'path': id_path, - 'callback': self._get, - }, - { - 'method': 'GET', - 'path': path + '/search/<search_criteria:path>', - 'callback': self._search, - }, - { - 'method': 'DELETE', - 'path': id_path, - 'callback': self._delete, - }, - ] - - def _create(self, content_type, body, *args, **kwargs): - if content_type != 'application/json': - raise qvarn.NotJson(content_type) - - if 'type' not in body: - body['type'] = self._coll.get_type_name() - - validator = qvarn.Validator() - try: - validator.validate_new_resource(body, self._coll.get_type()) - except qvarn.ValidationError as e: - qvarn.log.log('error', msg_text=str(e), body=body) - return qvarn.bad_request_response(str(e)) - - result_body = self._coll.post(body) - qvarn.log.log( - 'debug', msg_text='POST a new resource, result', - body=result_body) - location = '{}{}/{}'.format( - self._baseurl, self._coll.get_type().get_path(), result_body['id']) - self._notify(result_body['id'], result_body['revision'], 'created') - return qvarn.created_response(result_body, location) - - def _update(self, content_type, body, *args, **kwargs): - if content_type != 'application/json': - raise qvarn.NotJson(content_type) - - if 'type' not in body: - body['type'] = self._coll.get_type_name() - - if 'id' not in body: - body['id'] = kwargs['id'] - - validator = qvarn.Validator() - try: - validator.validate_resource_update(body, self._coll.get_type()) - except qvarn.ValidationError as e: - qvarn.log.log('error', msg_text=str(e), body=body) - return qvarn.bad_request_response(str(e)) - - obj_id = kwargs['id'] - # FIXME: the following test should be enabled once we - # no longer need test-api. - if False and body['id'] != obj_id: - raise qvarn.IdMismatch(body['id'], obj_id) - - try: - result_body = self._coll.put(body) - except qvarn.WrongRevision as e: - return qvarn.conflict_response(str(e)) - except qvarn.NoSuchResource as e: - # We intentionally say bad request, instead of not found. - # This is to be compatible with old Qvarn. This may get - # changed later. - return qvarn.bad_request_response(str(e)) - - self._notify(result_body['id'], result_body['revision'], 'updated') - return qvarn.ok_response(result_body) - - def _list(self, *args, **kwargs): - body = self._coll.list() - return qvarn.ok_response(body) - - def _get(self, *args, **kwargs): - try: - obj = self._coll.get(kwargs['id']) - except qvarn.NoSuchResource as e: - return qvarn.no_such_resource_response(str(e)) - return qvarn.ok_response(obj) - - def _search(self, *args, **kwargs): - path = kwargs['raw_uri_path'] - search_criteria = path.split('/search/', 1)[1] - try: - result = self._coll.search(search_criteria) - except qvarn.UnknownSearchField as e: - return qvarn.unknown_search_field_response(e) - except qvarn.NeedSortOperator: - return qvarn.need_sort_response() - except qvarn.SearchParserError as e: - return qvarn.search_parser_error_response(e) - body = { - 'resources': result, - } - return qvarn.ok_response(body) - - def _delete(self, *args, **kwargs): - obj_id = kwargs['id'] - self._coll.delete(obj_id) - self._notify(obj_id, None, 'deleted') - return qvarn.ok_response({}) diff --git a/qvarn/resource_type.py b/qvarn/resource_type.py deleted file mode 100644 index 40f8930..0000000 --- a/qvarn/resource_type.py +++ /dev/null @@ -1,150 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import os - -import yaml - - -class ResourceType: - - def __init__(self): - self._type = None - self._path = None - self._versions = [] - self._version = None - self._prototype = None - - def from_spec(self, spec): - self.set_type(spec['type']) - self.set_path(spec['path']) - self._versions = spec['versions'] - self.set_version_spec(self._versions[-1]) - - def as_dict(self): - return { - 'type': self.get_type(), - 'path': self.get_path(), - 'versions': [ - version_spec - for version_spec in self._versions - ] - } - - def set_type(self, type_name): - self._type = type_name - - def get_type(self): - return self._type - - def set_path(self, path): - self._path = path - - def get_path(self): - return self._path - - def set_version_spec(self, version_spec): - self._version = version_spec['version'] - self._prototype = version_spec['prototype'] - - def get_all_versions(self): - return [v['version'] for v in self._versions] - - def get_version(self, version): - for v in self._versions: - if v['version'] == version: - return v - raise KeyError('Version %r not found' % version) - - def get_latest_version(self): - return self._version - - def get_latest_prototype(self): - return self._prototype - - def get_subpaths(self): - v = self._versions[-1] - subpaths = v.get('subpaths', {}) - return { - subpath: subpaths[subpath]['prototype'] - for subpath in subpaths - } - - def get_subprototype(self, subpath): - v = self._versions[-1] - subpaths = v.get('subpaths', {}) - subproto = subpaths.get(subpath, {}) - return subproto.get('prototype') - - def get_files(self): - v = self._versions[-1] - return v.get('files', []) - - -def load_resource_types(dirname): # pragma: no cover - assert dirname is not None - resource_types = [] - basenames = [x for x in os.listdir(dirname) if x.endswith('.yaml')] - for basename in basenames: - pathname = os.path.join(dirname, basename) - with open(pathname) as f: - spec = yaml.safe_load(f) - rt = ResourceType() - rt.from_spec(spec) - resource_types.append(rt) - return resource_types - - -def add_missing_fields(proto, obj): - # Assume obj is validated. - - return _fill_in_dict(proto, obj) - - -def _fill_in_dict(proto, obj): - new = {} - defaults = { - str: '', - int: 0, - bool: False, - } - - for field in proto: - if type(proto[field]) in defaults: - if field not in obj: - new[field] = defaults[type(proto[field])] - elif isinstance(proto[field], list): - if field not in obj: - new[field] = [] - elif isinstance(proto[field][0], dict): - new[field] = [ - _fill_in_dict(proto[field][0], x) - for x in obj[field] - ] - elif type(proto[field][0]) in defaults: - new[field] = list(obj[field]) - else: # pragma: no cover - assert 0, 'field is {!r}'.format(field) - - if isinstance(obj, dict): # pragma: no cover - for field in obj: - if field not in new: - if isinstance(obj[field], list): - new[field] = list(obj[field]) - else: - new[field] = obj[field] - - return new diff --git a/qvarn/resource_type_tests.py b/qvarn/resource_type_tests.py deleted file mode 100644 index 36c775b..0000000 --- a/qvarn/resource_type_tests.py +++ /dev/null @@ -1,195 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - -import qvarn - - -class ResourceTypeTests(unittest.TestCase): - - def test_initially_has_no_type(self): - rt = qvarn.ResourceType() - self.assertEqual(rt.get_type(), None) - - def test_sets_type(self): - rt = qvarn.ResourceType() - rt.set_type('subject') - self.assertEqual(rt.get_type(), 'subject') - - def test_initially_has_no_path(self): - rt = qvarn.ResourceType() - self.assertEqual(rt.get_path(), None) - - def test_sets_path(self): - rt = qvarn.ResourceType() - rt.set_path('/subjects') - self.assertEqual(rt.get_path(), '/subjects') - - def test_initially_has_no_latest_version(self): - rt = qvarn.ResourceType() - self.assertEqual(rt.get_latest_version(), None) - - def test_load_resource_spec(self): - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'foo': '', - }, - }, - { - 'version': 'v1', - 'prototype': { - 'foo': '', - 'bar': '', - 'version': 0, # test for bug - }, - 'subpaths': { - 'subfoo': { - 'prototype': { - 'subbar': '', - }, - }, - 'blob': { - 'prototype': { - 'blob': 'blob', - 'content-type': '', - }, - }, - }, - 'files': [ - 'blob', - ], - }, - ], - } - rt = qvarn.ResourceType() - rt.from_spec(spec) - self.assertEqual(rt.get_type(), spec['type']) - self.assertEqual(rt.get_path(), spec['path']) - self.assertEqual(rt.get_all_versions(), ['v0', 'v1']) - self.assertEqual(rt.get_version('v0'), spec['versions'][0]) - self.assertEqual(rt.get_version('v1'), spec['versions'][1]) - with self.assertRaises(KeyError): - rt.get_version('v999') - self.assertEqual( - rt.get_latest_version(), spec['versions'][-1]['version']) - self.assertEqual( - rt.get_latest_prototype(), spec['versions'][-1]['prototype']) - self.assertEqual(rt.as_dict(), spec) - subpaths = spec['versions'][-1]['subpaths'] - self.assertEqual( - rt.get_subpaths(), - { - 'subfoo': subpaths['subfoo']['prototype'], - 'blob': subpaths['blob']['prototype'], - }, - ) - self.assertEqual( - rt.get_subprototype('subfoo'), - spec['versions'][-1]['subpaths']['subfoo']['prototype'] - ) - self.assertEqual(rt.get_files(), ['blob']) - self.assertEqual( - rt.get_subprototype('blob'), - { - 'blob': 'blob', - 'content-type': '', - } - ) - - -class AddMissingFieldsTests(unittest.TestCase): - - def setUp(self): - spec = { - 'type': 'subject', - 'path': '/subjects', - 'versions': [ - { - 'version': 'v1', - 'prototype': { - 'foo': '', - 'bars': [ - { - 'foobar': '', - 'yo': '', - 'names': [''], - }, - ], - }, - }, - ], - } - self.rt = qvarn.ResourceType() - self.rt.from_spec(spec) - self.proto = self.rt.get_latest_prototype() - - def test_fills_in_toplevel_fields(self): - self.assertEqual( - qvarn.add_missing_fields(self.proto, {}), - { - 'foo': '', - 'bars': [], - } - ) - - def test_fills_in_list_dict_fields(self): - obj = { - 'bars': [ - { - }, - ], - } - self.assertEqual( - qvarn.add_missing_fields(self.proto, obj), - { - 'foo': '', - 'bars': [ - { - 'foobar': '', - 'yo': '', - 'names': [], - }, - ], - } - ) - - def test_fills_in_list_of_strings(self): - obj = { - 'bars': [ - { - 'names': ['James Bond'], - }, - ], - } - self.assertEqual( - qvarn.add_missing_fields(self.proto, obj), - { - 'foo': '', - 'bars': [ - { - 'foobar': '', - 'yo': '', - 'names': ['James Bond'], - }, - ], - } - ) diff --git a/qvarn/schema.py b/qvarn/schema.py deleted file mode 100644 index 7e533b8..0000000 --- a/qvarn/schema.py +++ /dev/null @@ -1,71 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -def schema(r): - stack = [] - push(stack, [], r) - result = [] - for x in generate(stack): - result.append(x) - return result - - -def push(stack, name, r): - stack.append((name, r)) - - -def pop(stack): - return stack.pop() - - -def generate(stack): - funcs = { - dict: dict_schema, - list: list_schema, - str: simple_schema, - int: simple_schema, - bool: simple_schema, - type(None): simple_schema, - } - - while stack: - name, r = pop(stack) - for x in funcs[type(r)](stack, name, r): - yield x - - -def dict_schema(stack, name, r): - for key in reversed(sorted(r.keys())): - push(stack, name + [key], r[key]) - return [] # must return an iterable - - -def list_schema(stack, name, r): - qvarn.log.log( - 'trace', msg_text='list_schema', stack=stack, name=name, len_r=len(r)) - if len(r) > 0: - yield name, list, type(r[0]) - if isinstance(r[0], dict): - push(stack, name, r[0]) - else: - yield name, list, None - - -def simple_schema(stack, name, r): - yield name, type(r) diff --git a/qvarn/schema_tests.py b/qvarn/schema_tests.py deleted file mode 100644 index 10461ad..0000000 --- a/qvarn/schema_tests.py +++ /dev/null @@ -1,90 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest -import sys - - -import qvarn - - -class SchemaTests(unittest.TestCase): - - def test_generates_schema_for_simple_resource_type(self): - resource_type = { - 'type': '', - } - self.assertEqual( - qvarn.schema(resource_type), - [ - (['type'], str), - ] - ) - - def test_generates_schema_with_simple_list(self): - resource_type = { - 'type': '', - 'foos': [''], - } - self.assertEqual( - qvarn.schema(resource_type), - [ - (['foos'], list, str), - (['type'], str), - ] - ) - - def test_generates_schema_with_dict_list(self): - resource_type = { - 'type': '', - 'foos': [ - { - 'yos': '', - 'bars': [''], - }, - ], - } - self.assertEqual( - qvarn.schema(resource_type), - [ - (['foos'], list, dict), - (['foos', 'bars'], list, str), - (['foos', 'yos'], str), - (['type'], str), - ] - ) - - def test_generates_schema_from_deep_resource_type(self): - N = sys.getrecursionlimit() + 1 - resource_type = { - 'foos': '', - } - for _ in range(N): - resource_type = { - 'foos': [resource_type], - } - self.assertTrue(isinstance(qvarn.schema(resource_type), list)) - - def test_generates_schema_from_dict_with_empty_list(self): - resource_type = { - 'foos': [], - } - self.assertEqual( - qvarn.schema(resource_type), - [ - (['foos'], list, None), - ] - ) diff --git a/qvarn/search_parser.py b/qvarn/search_parser.py deleted file mode 100644 index 7a90f64..0000000 --- a/qvarn/search_parser.py +++ /dev/null @@ -1,150 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import urllib - -import qvarn - - -class SearchParser: - - conditions = { - 'contains': (2, qvarn.Contains), - 'exact': (2, qvarn.Equal), - 'ge': (2, qvarn.GreaterOrEqual), - 'gt': (2, qvarn.GreaterThan), - 'le': (2, qvarn.LessOrEqual), - 'lt': (2, qvarn.LessThan), - 'ne': (2, qvarn.NotEqual), - 'startswith': (2, qvarn.Startswith), - 'show': (1, None), - 'show_all': (0, None), - 'sort': (1, None), - 'offset': (1, None), - 'limit': (1, None), - } - - def parse(self, path): - if not path: - raise SearchParserError('No condition given') - - sp = qvarn.SearchParameters() - - pairs = list(self._parse_simple(path)) - - for operator, args in pairs: - if operator == 'show_all': - sp.set_show_all() - elif operator == 'show': - for field in args: - sp.add_show_field(field) - elif operator == 'sort': - for field in args: - sp.add_sort_key(field) - elif operator == 'offset': - sp.set_offset(int(args[0])) - elif operator == 'limit': - sp.set_limit(int(args[0])) - else: - klass = self.conditions[operator][1] - cond = klass(*args) - sp.add_cond(cond) - - self._check_params(sp) - - return sp - - def _parse_simple(self, path): - # Yield operator, args pairs. - words = [self._unquote(w) for w in path.split('/')] - assert len(words) > 0 - while words: - operator, words = words[0], words[1:] - if operator not in self.conditions: - raise SearchParserError( - 'Unknown condition {}'.format(operator)) - - num_args = self.conditions[operator][0] - if num_args > len(words): - raise SearchParserError( - 'Not enough args for {}'.format(operator)) - - args, words = words[:num_args], words[num_args:] - yield operator, args - - def _unquote(self, word): - return urllib.parse.unquote(word) - - def _check_params(self, sp): - has_sort = sp.sort_keys != [] - has_offset = sp.offset is not None - has_limit = sp.limit is not None - if (has_limit or has_offset) and not has_sort: - raise NeedSortOperator() - - -class SearchParserError(Exception): - - def __init__(self, msg): - super().__init__(self, msg) - - -class NeedSortOperator(SearchParserError): - - def __init__(self): - super().__init__('/offset and /limit only valid with /sort') - - -class SearchParameters: - - def __init__(self): - self.sort_keys = [] - self.show_fields = [] - self.show_all = False - self.cond = None - self.offset = None - self.limit = None - - def set_offset(self, offset): - if self.offset is not None: - raise SearchParserError('/offset may only be used once') - self.offset = offset - - def set_limit(self, limit): - if self.limit is not None: - raise SearchParserError('/limit may only be used once') - self.limit = limit - - def add_sort_key(self, field_name): - self.sort_keys.append(field_name) - - def add_show_field(self, field_name): - if self.show_all: - raise SearchParserError('/show_all and /show conflict') - self.show_fields.append(field_name) - - def set_show_all(self): - if self.show_fields: - raise SearchParserError('/show_all and /show conflict') - self.show_all = True - - def add_cond(self, cond): - if self.cond is None: - self.cond = cond - elif isinstance(self.cond, qvarn.All): - self.cond.append_subcondition(cond) - else: - self.cond = qvarn.All(self.cond, cond) diff --git a/qvarn/search_parser_tests.py b/qvarn/search_parser_tests.py deleted file mode 100644 index 9f45325..0000000 --- a/qvarn/search_parser_tests.py +++ /dev/null @@ -1,207 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - - -import qvarn - - -class SearchParserTests(unittest.TestCase): - - def test_raises_error_path_is_empty(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.SearchParserError): - p.parse('') - - def test_raises_error_if_condition_is_unknown(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.SearchParserError): - p.parse('unknown/foo/bar') - - def test_returns_exact_condition(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar') - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_fields, []) - self.assertEqual(sp.show_all, False) - self.assertTrue(isinstance(sp.cond, qvarn.Equal)) - self.assertEqual(sp.cond.name, 'foo') - self.assertEqual(sp.cond.pattern, 'bar') - - def test_handles_url_encoded_slash(self): - p = qvarn.SearchParser() - sp = p.parse('exact/operating_system/gnu%2Flinux') - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_fields, []) - self.assertEqual(sp.show_all, False) - self.assertTrue(isinstance(sp.cond, qvarn.Equal)) - self.assertEqual(sp.cond.name, 'operating_system') - self.assertEqual(sp.cond.pattern, 'gnu/linux') - - def test_raises_error_if_only_show_specified(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.SearchParserError): - p.parse('show') - - def test_raises_error_if_both_show_and_show_all_specified(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.SearchParserError): - p.parse('exact/foo/bar/show_all/show/foo') - - def test_raises_error_if_show_specified_but_without_operand(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.SearchParserError): - p.parse('exact/foo/bar/show') - - def test_returns_show_if_specified(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar/show/foo') - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_fields, ['foo']) - self.assertTrue(isinstance(sp.cond, qvarn.Equal)) - self.assertEqual(sp.cond.name, 'foo') - self.assertEqual(sp.cond.pattern, 'bar') - - def test_returns_show_all_if_specified(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar/show_all') - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_all, True) - self.assertTrue(isinstance(sp.cond, qvarn.Equal)) - self.assertEqual(sp.cond.name, 'foo') - self.assertEqual(sp.cond.pattern, 'bar') - - def test_returns_all_condition(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar/exact/foobar/yo') - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_fields, []) - self.assertTrue(isinstance(sp.cond, qvarn.All)) - self.assertEqual(len(sp.cond.conds), 2) - first, second = sp.cond.conds - self.assertTrue(isinstance(first, qvarn.Equal)) - self.assertEqual(first.name, 'foo') - self.assertEqual(first.pattern, 'bar') - self.assertTrue(isinstance(second, qvarn.Equal)) - self.assertEqual(second.name, 'foobar') - self.assertEqual(second.pattern, 'yo') - - def test_returns_sort_keys(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar/exact/foobar/yo/sort/a/sort/b') - self.assertEqual(sp.sort_keys, ['a', 'b']) - - def test_returns_sort_keys_with_show_all(self): - p = qvarn.SearchParser() - sp = p.parse('show_all/exact/foo/bar/exact/foobar/yo/sort/a/sort/b') - self.assertEqual(sp.sort_keys, ['a', 'b']) - - def test_sets_offset_and_limit(self): - p = qvarn.SearchParser() - sp = p.parse('exact/foo/bar/sort/a/offset/42/limit/128') - self.assertEqual(sp.offset, 42) - self.assertEqual(sp.limit, 128) - - def test_raises_error_for_offset_without_sort(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.NeedSortOperator): - p.parse('offset/1') - - def test_raises_error_for_limit_without_sort(self): - p = qvarn.SearchParser() - with self.assertRaises(qvarn.NeedSortOperator): - p.parse('limit/1') - - def test_accepts_limit_without_offset(self): - p = qvarn.SearchParser() - sp = p.parse('sort/x/limit/1') - self.assertEqual(sp.offset, None) - self.assertEqual(sp.limit, 1) - - -class SearchParametersTest(unittest.TestCase): - - def test_has_correct_initial_state(self): - sp = qvarn.SearchParameters() - self.assertEqual(sp.sort_keys, []) - self.assertEqual(sp.show_fields, []) - self.assertEqual(sp.show_all, False) - self.assertEqual(sp.cond, None) - - def test_sets_offset(self): - sp = qvarn.SearchParameters() - sp.set_offset(42) - self.assertEqual(sp.offset, 42) - - def test_raises_error_if_setting_offset_a_second_time(self): - sp = qvarn.SearchParameters() - sp.set_offset(42) - with self.assertRaises(qvarn.SearchParserError): - sp.set_offset(42) - - def test_sets_limit(self): - sp = qvarn.SearchParameters() - sp.set_limit(42) - self.assertEqual(sp.limit, 42) - - def test_raises_error_if_setting_limit_a_second_time(self): - sp = qvarn.SearchParameters() - sp.set_limit(42) - with self.assertRaises(qvarn.SearchParserError): - sp.set_limit(42) - - def test_adds_sort_key(self): - sp = qvarn.SearchParameters() - sp.add_sort_key('foo') - self.assertEqual(sp.sort_keys, ['foo']) - - def test_adds_show_field(self): - sp = qvarn.SearchParameters() - sp.add_show_field('foo') - self.assertEqual(sp.show_fields, ['foo']) - - def test_set_show_all(self): - sp = qvarn.SearchParameters() - sp.set_show_all() - self.assertEqual(sp.show_all, True) - - def test_show_when_show_all_is_set_raises_error(self): - sp = qvarn.SearchParameters() - sp.set_show_all() - with self.assertRaises(qvarn.SearchParserError): - sp.add_show_field('foo') - - def test_setting_show_all_when_fields_are_set_raises_error(self): - sp = qvarn.SearchParameters() - sp.add_show_field('foo') - with self.assertRaises(qvarn.SearchParserError): - sp.set_show_all() - - def test_adds_cond(self): - cond = qvarn.Yes() - sp = qvarn.SearchParameters() - self.assertEqual(sp.cond, None) - sp.add_cond(cond) - self.assertEqual(sp.cond, cond) - sp.add_cond(cond) - self.assertTrue(isinstance(sp.cond, qvarn.All)) - self.assertEqual(len(sp.cond.conds), 2) - self.assertEqual(sp.cond.conds, [cond, cond]) - sp.add_cond(cond) - self.assertTrue(isinstance(sp.cond, qvarn.All)) - self.assertEqual(len(sp.cond.conds), 3) - self.assertEqual(sp.cond.conds, [cond, cond, cond]) diff --git a/qvarn/sql.py b/qvarn/sql.py deleted file mode 100644 index 604272b..0000000 --- a/qvarn/sql.py +++ /dev/null @@ -1,339 +0,0 @@ -# sql.py - SQL dialect adaptation -# -# Copyright (C) 2017 Lars Wirzenius - - -'''Communicate with a PostgreSQL server.''' - - -import psycopg2 -import psycopg2.pool -import psycopg2.extras -import psycopg2.extensions - -import slog - -import qvarn - - -class PostgresAdapter: - - def __init__(self): - self._pool = None - - def connect(self, **kwargs): - self._pool = psycopg2.pool.ThreadedConnectionPool( - minconn=kwargs['min_conn'], - maxconn=kwargs['max_conn'], - database=kwargs['database'], - user=kwargs['user'], - password=kwargs['password'], - host=kwargs['host'], - port=kwargs['port'], - ) - - def transaction(self): - return Transaction(self) - - def get_conn(self): - return self._pool.getconn() - - def put_conn(self, conn): - self._pool.putconn(conn) - - -class Transaction: - - def __init__(self, sql): - self._sql = sql - self._conn = None - - def __enter__(self): - self._conn = self._sql.get_conn() - return self - - def __exit__(self, exc_type, exc_val, exc_tb): - try: - if exc_type is None: - self._conn.commit() - else: # pragma: no cover - self._conn.rollback() - except BaseException: # pragma: no cover - self._sql.put_conn(self._conn) - self._conn = None - raise - self._sql.put_conn(self._conn) - self._conn = None - - def execute(self, query, values): - qvarn.log.log('trace', msg_text='executing SQL query', query=query) - c = self._conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) - c.execute(query, values) - return c - - def get_rows(self, cursor): - for row in cursor: - yield dict(row) - - def create_table(self, table_name, **keys): - columns = ', '.join( - '{} {}'.format(self._q(key_name), self._sqltype(col_type)) - for key_name, col_type in keys.items() - ) - return 'CREATE TABLE IF NOT EXISTS {} ({})'.format( - self._q(table_name), - columns) - - def _sqltype(self, col_type): - types = [ - (str, 'TEXT'), - (int, 'BIGINT'), - (bool, 'BOOL'), - (dict, 'JSONB'), - (bytes, 'BYTEA'), - ] - for t, n in types: - if col_type == t: - return n - assert False - - def insert_object(self, table_name, *keys): - columns = [self._q(k) for k in keys] - placeholders = [self._placeholder(k) for k in keys] - return 'INSERT INTO {} ({}) VALUES ({})'.format( - self._q(table_name), - ', '.join(columns), - ', '.join(placeholders), - ) - - def remove_objects(self, table_name, *keys): - conditions = [ - '{} = {}'.format(self._q(key), self._placeholder(key)) - for key in keys - ] - return 'DELETE FROM {} WHERE {}'.format( - self._q(table_name), - ' AND '.join(conditions), - ) - - def select_objects(self, table_name, col_name, *keys): - conditions = [ - '{} = {}'.format(self._q(key), self._placeholder(key)) - for key in keys - ] - query = 'SELECT {} FROM {}'.format( - self._q(col_name), - self._q(table_name), - ) - if conditions: - query += ' WHERE {}'.format(' AND '.join(conditions)) - qvarn.log.log( - 'debug', msg_text='PostgresAdapater.select_objects', - query=query, keys=keys, conditions=conditions) - return query - - def select_objects_on_cond(self, table_name, cond, *keys): - query, values = qvarn.sql_select(_counter, cond) - return query, values - - def _condition(self, cond): - return cond.as_sql() - - def _q(self, name): - return quote(name) - - def _placeholder(self, name): - return placeholder(name) - - -def quote(name): - ascii_lower = 'abcdefghijklmnopqrstuvwxyz' - ascii_digits = '0123456789' - ascii_chars = ascii_lower + ascii_lower.upper() + ascii_digits - ok = ascii_chars + '_' - assert name.strip(ok) == '', 'must have only allowed chars: %r' % name - return '_'.join(name.split('-')) - - -def placeholder(name): - return '%({})s'.format(quote(name)) - - -_counter = slog.Counter() - - -def get_unique_name(base, counter=None): - if counter is None: - counter = _counter - return '{}{}'.format(base, counter.increment()) - - -class Condition: - - def get_subconditions(self): - return [] - - def matches(self, obj): # pragma: no cover - raise NotImplementedError() - - def as_sql(self): # pragma: no cover - raise NotImplementedError() - - -class All(Condition): - - def __init__(self, *conds): - self.conds = list(conds) - - def append_subcondition(self, cond): - self.conds.append(cond) - - def get_subconditions(self): - return self.conds - - def matches(self, obj): - for cond in self.conds: - if not cond.matches(obj): - return False - return True - - def as_sql(self): # pragma: no cover - pairs = [cond.as_sql() for cond in self.conds] - conds = ' AND '.join(query for query, _ in pairs) - values = {} - for _, value in pairs: - values.update(value) - return '( {} )'.format(conds), values - - -class Cmp(Condition): - - def __init__(self, name, pattern): - self.name = name - self.pattern = pattern - - def cmp_py(self, actual): - raise NotImplementedError() - - def cmp_sql(self, pattern_name): - return "lower(_field->>'value') {} lower(%({})s)".format( - self.get_operator(), pattern_name) - - def get_operator(self): - raise NotImplementedError() - - def matches(self, obj): - for key, actual in qvarn.flatten_object(obj): - if key == self.name and self.cmp_py(actual): - return True - return False - - def as_sql(self): # pragma: no cover - name_name = get_unique_name('name') - pattern_name = get_unique_name('pattern') - values = { - name_name: self.name, - pattern_name: self.pattern, - } - query = ("_field ->> 'name' = %%(%s)s AND " - "_field ->> 'value' %s") % ( - name_name, self.cmp_sql(pattern_name)) - return query, values - - -class Equal(Cmp): - - def cmp_py(self, actual): - return self.pattern.lower() == actual.lower() - - def get_operator(self): - return '=' - - -class ResourceTypeIs(Equal): - - def __init__(self, type_name): - super().__init__('type', type_name) - - def matches(self, obj): - return obj.get('type') == self.pattern - - -class NotEqual(Cmp): - - def cmp_py(self, actual): - return self.pattern.lower() != actual.lower() - - def get_operator(self): - return '!=' - - -class GreaterThan(Cmp): - - def cmp_py(self, actual): - return actual.lower() > self.pattern.lower() - - def get_operator(self): - return '>' - - -class GreaterOrEqual(Cmp): - - def cmp_py(self, actual): - return actual.lower() >= self.pattern.lower() - - def get_operator(self): - return '>=' - - -class LessThan(Cmp): - - def cmp_py(self, actual): - return actual.lower() < self.pattern.lower() - - def get_operator(self): - return '<' - - -class LessOrEqual(Cmp): - - def cmp_py(self, actual): - return actual.lower() <= self.pattern.lower() - - def get_operator(self): - return '<=' - - -class Contains(Cmp): - - def cmp_py(self, actual): - return self.pattern.lower() in actual.lower() - - def cmp_sql(self, pattern_name): - t = "lower(_field->>'value') LIKE '%%' || lower(%({})s) || '%%'" - return t.format(pattern_name) - - def get_operator(self): - pass - - -class Startswith(Cmp): - - def cmp_py(self, actual): - return actual.lower().startswith(self.pattern.lower()) - - def cmp_sql(self, pattern_name): - t = "lower(_field->>'value') LIKE lower(%({})s) || '%%'" - return t.format(pattern_name) - - def get_operator(self): - pass - - -class Yes(Condition): - - def matches(self, obj): - return True - - def as_sql(self): # pragma: no cover - return 'TRUE', {} diff --git a/qvarn/sql_select.py b/qvarn/sql_select.py deleted file mode 100644 index e54a5b9..0000000 --- a/qvarn/sql_select.py +++ /dev/null @@ -1,85 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -def sql_select(counter, cond): - conds = list(flatten(cond)) - if len(conds) == 1: - query, params = _select_on_simple_cond(counter, cond) - else: - query, params = _select_on_multiple_conds(counter, conds) - return query, params - - -def _select_on_simple_cond(counter, cond): - name = qvarn.get_unique_name('name', counter=counter) - value = qvarn.get_unique_name('value', counter=counter) - params = { - name: cond.name, - value: cond.pattern, - } - template = ' '.join(''' - SELECT _objects.obj_id, _objects.subpath, _objects._obj FROM _objects, - ( - SELECT obj_id FROM _aux WHERE - _field->>'name' = %({name})s - AND {valuecmp} - ) AS _temp - WHERE _temp.obj_id = _objects.obj_id - '''.split()) - query = template.format(name=name, valuecmp=cond.cmp_sql(value)) - return query, params - - -def _select_on_multiple_conds(counter, conds): - params = { - 'count': len(conds), - } - - part_template = "(_field->>'name' = %({name})s AND {valuecmp})" - parts = [] - for subcond in conds: - name = qvarn.get_unique_name('name', counter=counter) - value = qvarn.get_unique_name('value', counter=counter) - params[name] = subcond.name - params[value] = subcond.pattern - part = part_template.format(name=name, valuecmp=subcond.cmp_sql(value)) - parts.append(part) - - template = ' '.join(''' - SELECT _objects.obj_id, _objects.subpath, _objects._obj - FROM _objects, ( - SELECT obj_id, count(obj_id) AS _hits FROM _aux WHERE - {} - GROUP BY obj_id - ) AS _temp WHERE _hits >= %(count)s AND _temp.obj_id = _objects.obj_id - '''.split()) - - query = template.format(' OR '.join(parts)) - - return query, params - - -def flatten(cond): - subs = cond.get_subconditions() - if subs: - for sub in subs: - for c in flatten(sub): - yield c - else: - yield cond diff --git a/qvarn/sql_select_tests.py b/qvarn/sql_select_tests.py deleted file mode 100644 index 6cf3e56..0000000 --- a/qvarn/sql_select_tests.py +++ /dev/null @@ -1,96 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - -import slog - -import qvarn - - -class SqlSelectTests(unittest.TestCase): - - def test_returns_simple_query_for_simple_equal(self): - cond = qvarn.Equal('foo', 'bar') - counter = slog.Counter() - sql, values = qvarn.sql_select(counter, cond) - self.assertEqual( - sql, - ("SELECT _objects.obj_id, _objects.subpath, _objects._obj " - "FROM _objects, " - "( SELECT obj_id FROM _aux WHERE " - "_field->>'name' = %(name1)s AND " - "lower(_field->>'value') = lower(%(value2)s) ) AS _temp " - "WHERE _temp.obj_id = _objects.obj_id") - ) - self.assertEqual( - values, - { - 'name1': 'foo', - 'value2': 'bar', - } - ) - - def test_returns_simple_query_for_simple_not_equal(self): - cond = qvarn.NotEqual('foo', 'bar') - counter = slog.Counter() - sql, values = qvarn.sql_select(counter, cond) - self.assertEqual( - sql, - ("SELECT _objects.obj_id, _objects.subpath, _objects._obj " - "FROM _objects, " - "( SELECT obj_id FROM _aux WHERE " - "_field->>'name' = %(name1)s AND " - "lower(_field->>'value') != lower(%(value2)s) ) AS _temp " - "WHERE _temp.obj_id = _objects.obj_id") - ) - self.assertEqual( - values, - { - 'name1': 'foo', - 'value2': 'bar', - } - ) - - def test_returns_query_for_anded_conditions(self): - cond1 = qvarn.Equal('foo1', 'bar1') - cond2 = qvarn.NotEqual('foo2', 'bar2') - cond = qvarn.All(cond1, cond2) - counter = slog.Counter() - sql, values = qvarn.sql_select(counter, cond) - self.maxDiff = None - self.assertEqual( - sql, - ("SELECT _objects.obj_id, _objects.subpath, _objects._obj " - "FROM _objects, ( " - "SELECT obj_id, count(obj_id) AS _hits FROM _aux WHERE " - "(_field->>'name' = %(name1)s AND " - "lower(_field->>'value') = lower(%(value2)s)) OR " - "(_field->>'name' = %(name3)s AND " - "lower(_field->>'value') != lower(%(value4)s)) " - "GROUP BY obj_id ) AS _temp WHERE _hits >= %(count)s AND " - "_temp.obj_id = _objects.obj_id") - ) - self.assertEqual( - values, - { - 'name1': 'foo1', - 'value2': 'bar1', - 'name3': 'foo2', - 'value4': 'bar2', - 'count': 2, - } - ) diff --git a/qvarn/subresource_router.py b/qvarn/subresource_router.py deleted file mode 100644 index 343ee4a..0000000 --- a/qvarn/subresource_router.py +++ /dev/null @@ -1,82 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class SubresourceRouter(qvarn.Router): - - def __init__(self): - super().__init__() - self._parent_coll = None - self._subpath = None - - def set_subpath(self, subpath): - self._subpath = subpath - - def set_parent_collection(self, parent_coll): - self._parent_coll = parent_coll - - def get_routes(self): - rt = self._parent_coll.get_type() - path = '{}/<id>/{}'.format(rt.get_path(), self._subpath) - return [ - { - 'method': 'GET', - 'path': path, - 'callback': self._get_subresource, - }, - { - 'method': 'PUT', - 'path': path, - 'callback': self._put_subresource, - }, - ] - - def _get_subresource(self, *args, **kwargs): - obj_id = kwargs['id'] - try: - obj = self._parent_coll.get_subresource(obj_id, self._subpath) - except qvarn.NoSuchResource as e: - return qvarn.no_such_resource_response(str(e)) - return qvarn.ok_response(obj) - - def _put_subresource(self, content_type, body, *args, **kwargs): - if content_type != 'application/json': - raise qvarn.NotJson(content_type) - - obj_id = kwargs['id'] - if 'revision' not in body: - return qvarn.bad_request_response('must have revision') - revision = body.pop('revision') - - rt = self._parent_coll.get_type() - validator = qvarn.Validator() - try: - validator.validate_subresource(self._subpath, rt, body) - except qvarn.ValidationError as e: - qvarn.log.log('error', msg_text=str(e), body=body) - return qvarn.bad_request_response(str(e)) - - try: - result_body = self._parent_coll.put_subresource( - body, subpath=self._subpath, obj_id=obj_id, revision=revision) - except qvarn.WrongRevision as e: - return qvarn.conflict_response(str(e)) - except qvarn.NoSuchResource as e: - return qvarn.no_such_resource_response(str(e)) - - return qvarn.ok_response(result_body) diff --git a/qvarn/validator.py b/qvarn/validator.py deleted file mode 100644 index 725d947..0000000 --- a/qvarn/validator.py +++ /dev/null @@ -1,129 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import qvarn - - -class Validator: - - def validate_against_prototype( - self, resource_type_name, resource, prototype): - actual_schema = qvarn.schema(resource) - wanted_schema = qvarn.schema(prototype) - - allowed_names = [x[0] for x in wanted_schema] - for actual in actual_schema: - if actual[0] not in allowed_names: - dotted = '.'.join(actual[0]) - raise UnknownField(resource_type_name, dotted) - - def _validate(self, resource, resource_type): - if not isinstance(resource, dict): - raise NotADict(resource) - if 'type' not in resource: - raise NoType() - if resource['type'] != resource_type.get_type(): - raise WrongType(resource['type'], resource_type.get_type()) - - prototype = resource_type.get_latest_prototype() - self.validate_against_prototype(resource['type'], resource, prototype) - - def validate_new_resource(self, resource, resource_type): - self._validate(resource, resource_type) - if 'id' in resource: - raise HasId() - if 'revision' in resource: - raise HasRevision() - - def validate_resource_update(self, resource, resource_type): - self._validate(resource, resource_type) - if 'id' not in resource: - raise NoId() - if 'revision' not in resource: - raise NoRevision() - - def validate_subresource(self, subpath, resource_type, sub): - qvarn.log.log( - 'debug', msg_text='validating subresource', subpath=subpath, - sub=sub) - subproto = resource_type.get_subprototype(subpath) - if subproto is None: - raise UnknownSubpath(resource_type.get_type(), subpath) - self.validate_against_prototype('FIXME', sub, subproto) - - -class ValidationError(Exception): - - pass - - -class NotADict(ValidationError): # pragma: no cover - - def __init__(self, resource): - super().__init__('Was expecting a dict, got %r' % type(resource)) - - -class NoType(ValidationError): - - def __init__(self): - super().__init__("Resources MUST have a type field") - - -class WrongType(ValidationError): - - def __init__(self, actual, expected): - super().__init__( - 'Resource has type %s, but %s was expected' % (actual, expected)) - - -class NoId(ValidationError): - - def __init__(self): - super().__init__("PUTted resources MUST have an id set") - - -class HasId(ValidationError): - - def __init__(self): - super().__init__("POSTed resources MUST NOT have an id set") - - -class NoRevision(ValidationError): - - def __init__(self): - super().__init__("PUTted resources MUST have a revision set") - - -class HasRevision(ValidationError): - - def __init__(self): - super().__init__("POSTed resources MUST NOT have a revision set") - - -class UnknownField(ValidationError): # pragma: no cover - - def __init__(self, type_name, name): - super().__init__( - 'Resource type {} has unknown field {}'.format( - type_name, name)) - - -class UnknownSubpath(ValidationError): # pragma: no cover - - def __init__(self, type_name, subpath): - super().__init__( - 'Resource type {} has not sub-resource {}'.format( - type_name, subpath)) diff --git a/qvarn/validator_tests.py b/qvarn/validator_tests.py deleted file mode 100644 index 8322f89..0000000 --- a/qvarn/validator_tests.py +++ /dev/null @@ -1,131 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import unittest - - -import qvarn - - -class ValidatorTests(unittest.TestCase): - - def setUp(self): - self.spec = { - 'type': 'foo', - 'path': '/foos', - 'versions': [ - { - 'version': 'v0', - 'prototype': { - 'type': '', - 'id': '', - 'revision': '', - 'foo': '', - }, - 'subpaths': { - 'sub': { - 'prototype': { - 'subfield': '', - }, - }, - }, - }, - ] - } - - self.resource_type = qvarn.ResourceType() - self.resource_type.from_spec(self.spec) - - self.validator = qvarn.Validator() - - self.resource = { - 'type': 'foo', - 'id': 'resource-1', - 'revision': 'revision-1', - 'foo': 'this is foo', - } - - self.subresource = { - 'subfield': 'bananarama', - } - - def test_accepts_valid_resource(self): - self.validator.validate_resource_update( - self.resource, self.resource_type) - - def test_rejects_resource_that_is_not_a_dict(self): - with self.assertRaises(qvarn.NotADict): - self.validator.validate_new_resource(None, self.resource_type) - - def test_rejects_resource_without_type(self): - del self.resource['type'] - with self.assertRaises(qvarn.NoType): - self.validator.validate_new_resource( - self.resource, self.resource_type) - - def test_rejects_new_resource_with_different_type(self): - self.resource['type'] = 'wrong' - del self.resource['id'] - del self.resource['revision'] - with self.assertRaises(qvarn.WrongType): - self.validator.validate_new_resource( - self.resource, self.resource_type) - - def test_rejects_resource_update_with_different_type(self): - self.resource['type'] = 'wrong' - with self.assertRaises(qvarn.WrongType): - self.validator.validate_resource_update( - self.resource, self.resource_type) - - def test_rejects_resource_without_id(self): - del self.resource['id'] - with self.assertRaises(qvarn.NoId): - self.validator.validate_resource_update( - self.resource, self.resource_type) - - def test_rejects_resource_with_id(self): - with self.assertRaises(qvarn.HasId): - self.validator.validate_new_resource( - self.resource, self.resource_type) - - def test_rejects_resource_with_revision(self): - del self.resource['id'] - with self.assertRaises(qvarn.HasRevision): - self.validator.validate_new_resource( - self.resource, self.resource_type) - - def test_rejects_resource_without_revision(self): - del self.resource['revision'] - with self.assertRaises(qvarn.NoRevision): - self.validator.validate_resource_update( - self.resource, self.resource_type) - - def test_rejects_resource_with_unknown_field(self): - self.resource['unkown'] = '' - with self.assertRaises(qvarn.UnknownField): - self.validator.validate_resource_update( - self.resource, self.resource_type) - - def test_accepts_valid_subresource(self): - self.assertEqual( - self.validator.validate_subresource( - 'sub', self.resource_type, self.subresource), - None) - - def test_rejects_unknown_subresouce_path(self): - with self.assertRaises(qvarn.UnknownSubpath): - self.validator.validate_subresource( - 'unknown', self.resource_type, None) diff --git a/qvarn/version.py b/qvarn/version.py deleted file mode 100644 index bf504d8..0000000 --- a/qvarn/version.py +++ /dev/null @@ -1,2 +0,0 @@ -__version__ = "0.86+git" -__version_info__ = (0, 86, '+git') diff --git a/randport b/randport deleted file mode 100755 index d523401..0000000 --- a/randport +++ /dev/null @@ -1,28 +0,0 @@ -#!/usr/bin/python3 -# -# Copyright (C) 2017 Lars Wirzenius -# -# Pick a random port that is free to be listened on. For testing. - - -import errno -import random -import socket -import sys - - -MAX = 1000 -for i in range(MAX): - port = random.randint(1025, 2**15-1) - s = socket.socket() - try: - s.bind(('0.0.0.0', port)) - except OSError as e: - if e.errno == errno.EADDRINUSE: - continue - raise - break -else: - sys.stderr.write("Can't find a free port\n") - sys.exit(1) -sys.stdout.write('{}\n'.format(port)) diff --git a/resource_type/bolagsfakta_suppliers.yaml b/resource_type/bolagsfakta_suppliers.yaml deleted file mode 100644 index 8dda7b2..0000000 --- a/resource_type/bolagsfakta_suppliers.yaml +++ /dev/null @@ -1,17 +0,0 @@ -path: /bolagsfakta_suppliers -type: bolagsfakta_supplier -versions: -- prototype: - bolagsfakta_status: '' - contract_end_date: '' - contract_start_date: '' - id: '' - materialized_path: [''] - parent_org_id: '' - parent_supplier_id: '' - project_resource_id: '' - revision: '' - supplier_org_id: '' - supplier_type: '' - type: '' - version: v0 diff --git a/resource_type/cards.yaml b/resource_type/cards.yaml deleted file mode 100644 index 6fef7b7..0000000 --- a/resource_type/cards.yaml +++ /dev/null @@ -1,56 +0,0 @@ -path: /cards -type: card -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- files: [holder_photo, issuer_logo] - prototype: - card_ids: - - {card_id: '', card_id_type: ''} - card_status_history: - - {card_status: '', modification_description: '', modification_reason: '', modified_by: '', - modified_timestamp: ''} - card_type: '' - created_date: '' - current_status: '' - expiration_date: '' - holder_gov_ids: - - {country: '', gov_id: '', id_type: ''} - holder_names: - - full_name: '' - given_names: [''] - sort_key: '' - surnames: [''] - titles: [''] - holder_nationalities: [''] - id: '' - id06_supplier_full_name: '' - id06_taxation_country: '' - id06_virtual_bankid_required: false - id06_virtual_devices: [''] - id06_virtual_enabled: false - id06_virtual_history: - - {date: '', enabled: false, full_name: ''} - id06_virtual_valid_from: '' - id06_virtual_valid_until: '' - issued_date: '' - issuer_gov_org_ids: - - {country: '', gov_org_id: '', org_id_type: ''} - issuer_name: '' - org: '' - person: '' - revision: '' - type: '' - valid_from_date: '' - valid_until_date: '' - subpaths: - holder_photo: - prototype: {body: blob, content_type: ''} - issuer_logo: - prototype: {body: blob, content_type: ''} - sync: - prototype: - sync_revision: '' - sync_sources: - - {sync_id: '', sync_source: ''} - version: v1 diff --git a/resource_type/competence_types.yaml b/resource_type/competence_types.yaml deleted file mode 100644 index 5bc003c..0000000 --- a/resource_type/competence_types.yaml +++ /dev/null @@ -1,107 +0,0 @@ -path: /competence_types -type: competence_type -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: &id001 - competence_type_id: '' - descriptions: - - {description: '', locale: ''} - id: '' - names: - - {locale: '', name: ''} - revision: '' - type: '' - version: v1 -- files: [card_front, card_back, registry_logo] - prototype: *id001 - subpaths: - card_back: - prototype: &id002 {body: blob, content_type: ''} - card_front: - prototype: &id003 {body: blob, content_type: ''} - registry_logo: - prototype: &id004 {body: blob, content_type: ''} - version: v2 -- files: [card_front, card_back, registry_logo] - prototype: - competence_type_id: '' - customer_org_ids: [''] - descriptions: - - {description: '', locale: ''} - id: '' - names: - - {locale: '', name: ''} - revision: '' - type: '' - subpaths: - card_back: - prototype: *id002 - card_front: - prototype: *id003 - registry_logo: - prototype: *id004 - version: v3 -- files: [card_front, card_back, registry_logo] - prototype: - competence_type_id: '' - customer_org_ids: [''] - descriptions: - - {description: '', locale: ''} - id: '' - master_register_name: '' - master_register_org_id: '' - names: - - {locale: '', name: ''} - revision: '' - type: '' - subpaths: - card_back: - prototype: *id002 - card_front: - prototype: *id003 - registry_logo: - prototype: *id004 - version: v4 -- files: [card_front, card_back, registry_logo] - prototype: - competence_type_id: '' - customer_org_ids: [''] - descriptions: - - {description: '', locale: ''} - id: '' - is_verified: false - names: - - {locale: '', name: ''} - revision: '' - type: '' - subpaths: - card_back: - prototype: *id002 - card_front: - prototype: *id003 - registry_logo: - prototype: *id004 - version: v5 -- files: [card_front, card_back, registry_logo] - prototype: - attribute_validation: - - {attribute: '', regex: '', required: false} - competence_type_id: '' - customer_org_ids: [''] - descriptions: - - {description: '', locale: ''} - id: '' - is_verified: false - names: - - {locale: '', name: ''} - revision: '' - type: '' - subpaths: - card_back: - prototype: *id002 - card_front: - prototype: *id003 - registry_logo: - prototype: *id004 - version: v6 diff --git a/resource_type/competences.yaml b/resource_type/competences.yaml deleted file mode 100644 index 9f6a301..0000000 --- a/resource_type/competences.yaml +++ /dev/null @@ -1,47 +0,0 @@ -path: /competences -type: competence -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: - competence_card_holder_names: - - full_name: '' - given_names: [''] - sort_key: '' - surnames: [''] - competence_card_id: '' - competence_type_id: '' - employee_person_id: '' - employer_org_id: '' - id: '' - revision: '' - type: '' - valid_from_date: '' - valid_until_date: '' - subpaths: - sync: - prototype: &id001 {sync_id: '', sync_revision: ''} - version: v1 -- prototype: - competence_card_holder_names: - - full_name: '' - given_names: [''] - sort_key: '' - surnames: [''] - competence_card_id: '' - competence_registry_id: '' - competence_trainer: '' - competence_type_id: '' - employee_person_id: '' - employer_org_id: '' - granted_by_org_id: '' - id: '' - revision: '' - type: '' - valid_from_date: '' - valid_until_date: '' - validation_status: '' - subpaths: - sync: - prototype: *id001 - version: v2 diff --git a/resource_type/contracts.yaml b/resource_type/contracts.yaml deleted file mode 100644 index 6cf8bc1..0000000 --- a/resource_type/contracts.yaml +++ /dev/null @@ -1,416 +0,0 @@ -path: /contracts -type: contract -versions: - -- version: v0 - prototype: - id: '' - revision: '' - type: '' - -- version: v1 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v2 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_roles: '' - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v3 - prototype: - contract_parties: - - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v4 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_roles: '' - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v5 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_roles: '' - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - global_permissions: - - permission_name: '' - permissions: - - permission_name: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v6 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_roles: '' - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - global_permissions: - - permission_name: '' - permissions: - - permission_name: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - terms_of_service: - - acceptance_time: '' - accepter_person_id: '' - terms_of_service_version: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document - -- version: v7 - prototype: - contract_parties: - - contacts: - - address_lines: [''] - contact_roles: '' - contact_source: '' - contact_timestamp: '' - contact_type: '' - country: '' - email_address: '' - full_address: '' - phone_number: '' - post_area: '' - post_code: '' - global_permissions: - - permission_name: '' - permissions: - - permission_name: '' - resource_id: '' - role: '' - type: '' - user_role: '' - username: '' - contract_state: '' - contract_state_history: - - modification_timestamp: '' - modified_by: '' - state: '' - contract_type: '' - date_created: '' - date_updated: '' - end_date: '' - id: '' - id06_contact_email: '' - id06_contact_name: '' - id06_issuer_requires_bankid: false - preferred_language: '' - revision: '' - right_to_work_based_on: '' - signers: - - signer: '' - signing_request_message: '' - signing_request_timestamp: '' - signing_timestamp: '' - start_date: '' - terms_of_service: - - acceptance_time: '' - accepter_person_id: '' - terms_of_service_language: '' - terms_of_service_version: '' - type: '' - subpaths: - document: - prototype: - body: blob - content_type: '' - sync: - prototype: - sync_revision: '' - sync_sources: - - sync_id: '' - sync_source: '' - files: - - document diff --git a/resource_type/data_cache.yaml b/resource_type/data_cache.yaml deleted file mode 100644 index d1574b7..0000000 --- a/resource_type/data_cache.yaml +++ /dev/null @@ -1,8 +0,0 @@ -path: /data_cache -type: data_cache -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: {cache_name: '', expires_at: '', id: '', key: '', revision: '', type: '', - value: ''} - version: v1 diff --git a/resource_type/events.yaml b/resource_type/events.yaml deleted file mode 100644 index 589ac85..0000000 --- a/resource_type/events.yaml +++ /dev/null @@ -1,14 +0,0 @@ -path: /events -type: event -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: {card: '', card_event_type: '', event_type: '', generated_timestamp: '', - id: '', org: '', person: '', project: '', revision: '', type: ''} - subpaths: - sync: - prototype: - sync_revision: '' - sync_sources: - - {sync_id: '', sync_source: ''} - version: v1 diff --git a/resource_type/files.yaml b/resource_type/files.yaml deleted file mode 100644 index ef74217..0000000 --- a/resource_type/files.yaml +++ /dev/null @@ -1,16 +0,0 @@ -path: /files -type: file -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- files: [file] - prototype: - filename: '' - id: '' - revision: '' - type: '' - version: 0 - subpaths: - file: - prototype: {body: blob, content_type: ''} - version: v1 diff --git a/resource_type/jobs.yaml b/resource_type/jobs.yaml deleted file mode 100644 index a4222b0..0000000 --- a/resource_type/jobs.yaml +++ /dev/null @@ -1,20 +0,0 @@ -path: /jobs -type: job -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: - done_at: '' - id: '' - job_type: '' - org_id: '' - parameters: - - {key: '', value: ''} - person_id: '' - reserved_until: '' - revision: '' - started_at: '' - status: '' - submitted_at: '' - type: '' - version: v1 diff --git a/resource_type/listeners.yaml b/resource_type/listeners.yaml deleted file mode 100644 index 19caa3f..0000000 --- a/resource_type/listeners.yaml +++ /dev/null @@ -1,12 +0,0 @@ -type: listener -path: /listeners -versions: - - version: v0 - prototype: - id: "" - type: "" - revision: "" - listen_on_type: "" - notify_of_new: false - listen_on_all: false - listen_on: [""] diff --git a/resource_type/notifications.yaml b/resource_type/notifications.yaml deleted file mode 100644 index 9660282..0000000 --- a/resource_type/notifications.yaml +++ /dev/null @@ -1,13 +0,0 @@ -type: notification -path: /notifications -versions: - - version: v0 - prototype: - id: "" - type: "" - revision: "" - listener_id: "" - resource_id: "" - resource_revision: "" - resource_change: "" - timestamp: "" diff --git a/resource_type/orgs.yaml b/resource_type/orgs.yaml deleted file mode 100644 index 6566f80..0000000 --- a/resource_type/orgs.yaml +++ /dev/null @@ -1,140 +0,0 @@ -type: org -path: /orgs -versions: - -- version: v0 - prototype: - type: "" - id: "" - revision: "" - -- version: v1 - prototype: - type: "" - id: "" - revision: "" - country: "" - names: [""] - gov_org_ids: - - country: "" - org_id_type: "" - gov_org_id: "" - contacts: - - contact_type: "" - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - post_code: "" - post_area: "" - is_luotettava_kumppani_member: False - subpaths: - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - -- version: v2 - prototype: - type: "" - id: "" - revision: "" - country: "" - names: [""] - gov_org_ids: - - country: "" - org_id_type: "" - gov_org_id: "" - contacts: - - contact_type: "" - contact_roles: [""] - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - post_code: "" - post_area: "" - is_luotettava_kumppani_member: False - subpaths: - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - -- version: v3 - prototype: - type: "" - id: "" - revision: "" - country: "" - names: [""] - gov_org_ids: - - country: "" - org_id_type: "" - gov_org_id: "" - contacts: - - contact_type: "" - contact_roles: [""] - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - post_code: "" - post_area: "" - einvoice_operator: "" - einvoice_address: "" - is_luotettava_kumppani_member: False - subpaths: - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - -- version: v4 - prototype: - type: "" - id: "" - revision: "" - country: "" - names: [""] - gov_org_ids: - - country: "" - org_id_type: "" - gov_org_id: "" - contacts: - - contact_type: "" - contact_roles: [""] - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - post_code: "" - post_area: "" - einvoice_operator: "" - einvoice_address: "" - is_luotettava_kumppani_member: False - subpaths: - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" diff --git a/resource_type/persons.yaml b/resource_type/persons.yaml deleted file mode 100644 index 7986238..0000000 --- a/resource_type/persons.yaml +++ /dev/null @@ -1,164 +0,0 @@ -type: person -path: /persons -versions: - -- version: v0 - prototype: - type: "" - id: "" - revision: "" - -- version: v1 - prototype: - type: "" - id: "" - revision: "" - names: - - full_name: "" - sort_key: "" - titles: [""] - given_names: [""] - surnames: [""] - subpaths: - photo: - prototype: - body: blob - content_type: "" - private: - prototype: - date_of_birth: "" - gov_ids: - - country: "" - id_type: "" - gov_id: "" - contacts: - - contact_type: "" - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - post_code: "" - post_area: "" - verification_code: "" - verification_code_expiration_date: "" - email_verification_timestamp: "" - nationalities: [""] - residences: - - country: "" - location: "" - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - files: - - photo - -- version: v2 - prototype: - type: "" - id: "" - revision: "" - names: - - full_name: "" - sort_key: "" - titles: [""] - given_names: [""] - surnames: [""] - subpaths: - photo: - prototype: - body: blob - content_type: "" - private: - prototype: - date_of_birth: "" - gov_ids: - - country: "" - id_type: "" - gov_id: "" - contacts: - - contact_type: "" - contact_roles: [""] - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - - post_code: "" - post_area: "" - verification_code: "" - verification_code_expiration_date: "" - email_verification_timestamp: "" - nationalities: [""] - residences: - - country: "" - location: "" - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - files: - - photo - -- version: v3 - prototype: - type: "" - id: "" - revision: "" - gluu_user_id: "" - names: - - full_name: "" - sort_key: "" - titles: [""] - given_names: [""] - surnames: [""] - subpaths: - photo: - prototype: - body: blob - content_type: "" - private: - prototype: - date_of_birth: "" - gov_ids: - - country: "" - id_type: "" - gov_id: "" - contacts: - - contact_type: "" - contact_roles: [""] - contact_source: "" - contact_timestamp: "" - phone_number: "" - email_address: "" - full_address: "" - country: "" - address_lines: [""] - - post_code: "" - post_area: "" - verification_code: "" - verification_code_expiration_date: "" - email_verification_timestamp: "" - nationalities: [""] - residences: - - country: "" - location: "" - sync: - prototype: - sync_sources: - - sync_source: "" - sync_id: "" - sync_revision: "" - files: - - photo diff --git a/resource_type/projects.yaml b/resource_type/projects.yaml deleted file mode 100644 index db62932..0000000 --- a/resource_type/projects.yaml +++ /dev/null @@ -1,21 +0,0 @@ -path: /projects -type: project -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: - id: '' - names: [''] - project_ids: - - {project_id: '', project_id_type: ''} - project_responsible_org: '' - project_responsible_person: '' - revision: '' - type: '' - subpaths: - sync: - prototype: - sync_revision: '' - sync_sources: - - {sync_id: '', sync_source: ''} - version: v1 diff --git a/resource_type/report_accesses.yaml b/resource_type/report_accesses.yaml deleted file mode 100644 index 394b2c2..0000000 --- a/resource_type/report_accesses.yaml +++ /dev/null @@ -1,8 +0,0 @@ -path: /report_accesses -type: report_access -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- prototype: {access_time: '', arkisto_id: '', client_id: '', customer_id: '', id: '', - org_id: '', report_id: '', revision: '', type: ''} - version: v1 diff --git a/resource_type/reports.yaml b/resource_type/reports.yaml deleted file mode 100644 index 4163797..0000000 --- a/resource_type/reports.yaml +++ /dev/null @@ -1,17 +0,0 @@ -path: /reports -type: report -versions: -- prototype: {id: '', revision: '', type: ''} - version: v0 -- files: [pdf] - prototype: {generated_timestamp: '', id: '', org: '', report_type: '', revision: '', - tilaajavastuu_status: '', type: ''} - subpaths: - pdf: - prototype: {body: blob, content_type: ''} - sync: - prototype: - sync_revision: '' - sync_sources: - - {sync_id: '', sync_source: ''} - version: v1 diff --git a/resource_type/resource_types.yaml b/resource_type/resource_types.yaml deleted file mode 100644 index 45f5af5..0000000 --- a/resource_type/resource_types.yaml +++ /dev/null @@ -1,11 +0,0 @@ -type: resource_type -path: /resource_types -versions: - -- version: v0 - prototype: - type: "" - id: "" - revision: "" - name: "" - yaml: "" diff --git a/run-qvarn-debug b/run-qvarn-debug deleted file mode 100755 index e272227..0000000 --- a/run-qvarn-debug +++ /dev/null @@ -1,186 +0,0 @@ -#!/bin/sh -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - -set -eu - -default_scopes=" -uapi_persons_get -uapi_persons_id_get -uapi_persons_post -uapi_subjects_get -uapi_subjects_id_get -uapi_subjects_post -uapi_version_get - -uapi_events_post -uapi_events_get -uapi_events_id_get -uapi_events_id_put -uapi_events_id_delete -uapi_files_post -uapi_files_get -uapi_files_id_get -uapi_files_search_id_get -uapi_files_id_put -uapi_files_id_delete -uapi_files_id_file_get -uapi_files_id_file_put -uapi_jobs_post -uapi_jobs_get -uapi_jobs_id_get -uapi_jobs_search_id_get -uapi_jobs_id_put -uapi_jobs_id_delete -uapi_contracts_post -uapi_contracts_get -uapi_contracts_id_get -uapi_contracts_search_id_get -uapi_contracts_id_put -uapi_contracts_id_delete -uapi_contracts_id_document_get -uapi_contracts_id_document_put -uapi_persons_post -uapi_persons_get -uapi_persons_id_get -uapi_persons_search_id_get -uapi_persons_id_put -uapi_persons_id_delete -uapi_persons_id_private_get -uapi_persons_id_private_put -uapi_persons_id_photo_get -uapi_persons_id_photo_put -uapi_report_accesses_post -uapi_report_accesses_get -uapi_report_accesses_id_get -uapi_report_accesses_search_id_get -uapi_report_accesses_id_put -uapi_report_accesses_id_delete -uapi_report_accesses_id_pdf_get -uapi_report_accesses_id_pdf_put -uapi_data_cache_post -uapi_data_cache_get -uapi_data_cache_id_get -uapi_data_cache_search_id_get -uapi_data_cache_id_put -uapi_data_cache_id_delete -uapi_projects_post -uapi_projects_get -uapi_projects_id_get -uapi_projects_search_id_get -uapi_projects_id_put -uapi_projects_id_delete -uapi_competence_types_post -uapi_competence_types_get -uapi_competence_types_id_get -uapi_competence_types_search_id_get -uapi_competence_types_id_put -uapi_competence_types_id_delete -uapi_competence_types_id_card_front_get -uapi_competence_types_id_card_front_put -uapi_competence_types_id_card_back_get -uapi_competence_types_id_card_back_put -uapi_competence_types_id_registry_logo_get -uapi_competence_types_id_registry_logo_put -uapi_reports_post -uapi_reports_get -uapi_reports_id_get -uapi_reports_search_id_get -uapi_reports_id_put -uapi_reports_id_delete -uapi_reports_id_pdf_get -uapi_reports_id_pdf_put -uapi_competences_post -uapi_competences_get -uapi_competences_id_get -uapi_competences_search_id_get -uapi_competences_id_put -uapi_competences_id_delete -uapi_version_post -uapi_version_get -uapi_version_id_get -uapi_version_search_id_get -uapi_version_id_put -uapi_version_id_delete -uapi_bolagsfakta_suppliers_post -uapi_bolagsfakta_suppliers_get -uapi_bolagsfakta_suppliers_id_get -uapi_bolagsfakta_suppliers_search_id_get -uapi_bolagsfakta_suppliers_id_put -uapi_bolagsfakta_suppliers_id_delete -uapi_cards_post -uapi_cards_get -uapi_cards_id_get -uapi_cards_search_id_get -uapi_cards_id_put -uapi_cards_id_delete -uapi_cards_id_holder_photo_get -uapi_cards_id_holder_photo_put -uapi_cards_id_issuer_logo_get -uapi_cards_id_issuer_logo_put -uapi_orgs_post -uapi_orgs_get -uapi_orgs_id_get -uapi_orgs_search_id_get -uapi_orgs_id_put -uapi_orgs_id_delete -" - -tmp="$(mktemp -d)" - -cleanup() -{ - rm -rf "$tmp" -} - -trap cleanup EXIT - -token="$1" -case "$#" in - 2) key="$2" ;; - 1) key="$tmp/key" ;; -esac - - - -ISS=test -AUD=aud -./generate-rsa-key "$key" -./create-token "$key" "$ISS" "$AUD" "$default_scopes" > "$token" - -cat <<EOF > "$tmp/qvarn.yaml" -baseurl: http://localhost:12765 -log: - - filename: qvarn.log -token-issuer: $ISS -token-audience: $AUD -token-public-key: $(cat "$key.pub") -resource-type-dir: resource_type -memory-database: yes -database: - host: localhost - port: 5432 - database: qvarn - user: qvarn - password: pass - min_conn: 1 - max_conn: 1 -EOF -export QVARN_CONFIG="$tmp/qvarn.yaml" - -export PYTHONPATH=/home/liw/apifw -gunicorn3 --bind 127.0.0.1:12765 -p "$tmp/pid" -w1 --log-file g.log \ - --log-level debug \ - qvarn.backend:app diff --git a/qvarn/timestamp.py b/salami/__init__.py index 355bee0..f3b31d6 100644 --- a/qvarn/timestamp.py +++ b/salami/__init__.py @@ -14,12 +14,13 @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. -import time - - -def get_current_timestamp(): - t = time.time() - tm = time.gmtime(t) - ss = t - int(t) - secs = '%f' % ss - return time.strftime('%Y-%m-%dT%H:%M:%S', tm) + secs[1:] +from .version import __version__, __version_info__ +from .responses import ( + bad_request_response, + created_response, + ok_response, +) +from .log_setup import setup_logging, log +from .router import Router +from .version_router import VersionRouter +from .api import SalamiAPI diff --git a/salami/api.py b/salami/api.py new file mode 100644 index 0000000..ce839d9 --- /dev/null +++ b/salami/api.py @@ -0,0 +1,31 @@ +# Copyright (C) 2017 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, either version 3 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + + +import salami + + +class SalamiAPI: + + def __init__(self): + pass + + def find_missing_route(self, path): + salami.log.log('info', msg_text='find_missing_route', path=path) + + if path == '/version': + salami.log.log('info', msg_text='Add /version route') + v = salami.VersionRouter() + return v.get_routes() diff --git a/salami/backend.py b/salami/backend.py new file mode 100644 index 0000000..6412cb3 --- /dev/null +++ b/salami/backend.py @@ -0,0 +1,78 @@ +#!/usr/bin/python3 +# Copyright (C) 2017 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, either version 3 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + + +import os + +import apifw +import slog +import yaml + +import salami + + +DEFAULT_CONFIG_FILE = '/dev/null' + + +def dict_logger(log, stack_info=None): + salami.log.log(exc_info=stack_info, **log) + + +def read_config(filename): + with open(filename) as f: + return yaml.safe_load(f) + + +def check_config(cfg): + for key in cfg: + if cfg[key] is None: + raise Exception('Configration %s should not be None' % key) + + +_counter = slog.Counter() + + +def counter(): + new_context = 'HTTP transaction {}'.format(_counter.increment()) + salami.log.set_context(new_context) + + +default_config = { + 'token-public-key': None, + 'token-audience': None, + 'token-issuer': None, + 'log': [], +} + + +config_filename = os.environ.get('SALAMI_CONFIG', DEFAULT_CONFIG_FILE) +actual_config = read_config(config_filename) +config = dict(default_config) +config.update(actual_config or {}) +check_config(config) +salami.setup_logging(config) +salami.log.log('info', msg_text='Salami starting') + +api = salami.SalamiAPI() +app = apifw.create_bottle_application(api, counter, dict_logger, config) + +# If we are running this program directly with Python, and not via +# gunicorn, we can use the Bottle built-in debug server, which can +# make some things easier to debug. + +if __name__ == '__main__': + print('running in debug mode') + app.run(host='127.0.0.1', port=12765) diff --git a/qvarn/logging.py b/salami/log_setup.py index 67f3350..e32137c 100644 --- a/qvarn/logging.py +++ b/salami/log_setup.py @@ -43,21 +43,7 @@ slog.hijack_logging(log, logger_names=gunicorn_loggers) def setup_logging(config): for target in config.get('log', []): - setup_logging_to_target(target) - - -def setup_logging_to_target(target): - rule = get_filter_rules(target) - if 'filename' in target: - setup_logging_to_file(target, rule) - else: - raise Exception('Do not understand logging target %r' % target) - - -def get_filter_rules(target): - if 'filter' in target: - return slog.construct_log_filter(target['filter']) - return slog.FilterAllow() + setup_logging_to_file(target, slog.FilterAllow()) def setup_logging_to_file(target, rule): diff --git a/qvarn/responses.py b/salami/responses.py index eea78d7..138ba28 100644 --- a/qvarn/responses.py +++ b/salami/responses.py @@ -37,10 +37,6 @@ def ok_response(body, headers=None): return response(apifw.HTTP_OK, body, headers) -def no_such_resource_response(msg): - return response(apifw.HTTP_NOT_FOUND, msg, {}) - - def created_response(body, location): headers = { 'Content-Type': 'application/json', @@ -54,44 +50,3 @@ def bad_request_response(body): 'Content-Type': 'text/plain', } return response(apifw.HTTP_BAD_REQUEST, body, headers) - - -def need_sort_response(): - headers = { - 'Content-Type': 'application/json', - } - body = { - 'message': 'LIMIT and OFFSET can only be used with together SORT.', - 'error_code': 'LimitWithoutSortError', - } - return response(apifw.HTTP_BAD_REQUEST, body, headers) - - -def search_parser_error_response(e): - headers = { - 'Content-Type': 'application/json', - } - body = { - 'message': 'Could not parse search condition', - 'error_code': 'BadSearchCondition', - } - return response(apifw.HTTP_BAD_REQUEST, body, headers) - - -def unknown_search_field_response(e): - headers = { - 'Content-Type': 'application/json', - } - body = { - 'field': e.field, - 'message': 'Resource does not contain given field', - 'error_code': 'FieldNotInResource', - } - return response(apifw.HTTP_BAD_REQUEST, body, headers) - - -def conflict_response(body): - headers = { - 'Content-Type': 'text/plain', - } - return response(apifw.HTTP_CONFLICT, body, headers) diff --git a/qvarn/router.py b/salami/router.py index 9f171b0..9f171b0 100644 --- a/qvarn/router.py +++ b/salami/router.py diff --git a/salami/version.py b/salami/version.py new file mode 100644 index 0000000..4e6afe5 --- /dev/null +++ b/salami/version.py @@ -0,0 +1,2 @@ +__version__ = "0.0+git" +__version_info__ = (0, 0, '+git') diff --git a/qvarn/version_router.py b/salami/version_router.py index b9f1056..bc60d5c 100644 --- a/qvarn/version_router.py +++ b/salami/version_router.py @@ -14,10 +14,10 @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. -import qvarn +import salami -class VersionRouter(qvarn.Router): +class VersionRouter(salami.Router): def get_routes(self): return [ @@ -31,12 +31,6 @@ class VersionRouter(qvarn.Router): def _version(self, *args, **kwargs): version = { - 'api': { - 'version': qvarn.__version__, - }, - 'implementation': { - 'name': 'Qvarn', - 'version': qvarn.__version__, - }, + 'version': salami.__version__, } - return qvarn.ok_response(version) + return salami.ok_response(version) diff --git a/scripts/build-and-deploy b/scripts/build-and-deploy deleted file mode 100755 index f26f250..0000000 --- a/scripts/build-and-deploy +++ /dev/null @@ -1,76 +0,0 @@ -#!/bin/sh -# -# scripts/build-and-deploy - build Qvarn and upgrade it on given host -# -# Copyright 2016-2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - -# This program builds Qvarn and installs it on the host given on the -# command line as an upgrade. The host must already have been -# configured to run Qvarn. This script does nothing to update -# configuration files, for example. -# -# This script is meant to be used while developing Qvarn: make -# changes, commit them, build, deploy to a server, and then test the -# server. -# -# Usage: -# -# $0 HOST BOOL -# -# where HOST is the address of the server on which to upgrade Qvarn, -# and BOOL is "true" to indicate the databases should be emptied -# (recreated), and log files deleted, and "false" if not. -# -# Note that changes must be committed to git before running this -# script, and you must be able to ssh into the server and run sudo -# there. - - -set -eux - -[ -d .git ] -[ -e qvarn/__init__.py ] - -host="$1" -cleandb="$2" - -debian_version="$(dpkg-parsechangelog | sed -n '/^Version: /s///p')" -upstream_version="$(echo "$debian_version" | sed 's/-.*//')" -package="qvarn-jsonb" - -rm -f ../qvarn*_* - -git archive HEAD | xz > "../${package}_${upstream_version}.orig.tar.xz" - -if [ -f "/etc/debian_version" ]; then - debuild -us -uc -else - debuild -d -us -uc -fi - -deb="${package}_${debian_version}_all.deb" -scp "../$deb" "$host:" - -ssh "$host" sudo systemctl stop qvarn || true -ssh "$host" sudo dpkg -i "$deb" -if [ "$cleandb" = true ] -then - ssh "$host" sudo -u postgres dropdb qvarn || true - ssh "$host" sudo -u postgres createdb -O qvarn -E UTF8 -T template0 qvarn - ssh "$host" sudo find /var/log/qvarn -mindepth 1 -type f -delete -fi -ssh "$host" sudo systemctl restart haproxy -ssh "$host" sudo systemctl restart qvarn diff --git a/scripts/pgdump b/scripts/pgdump deleted file mode 100755 index 61b2994..0000000 --- a/scripts/pgdump +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/sh - -set -eu - -sudo -u postgres pg_dump qvarn diff --git a/scripts/pgempty b/scripts/pgempty deleted file mode 100755 index f0962a0..0000000 --- a/scripts/pgempty +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/sh - -set -eu - -sudo -u postgres dropdb qvarn || true -sudo -u postgres createdb -O qvarn -E UTF8 -T template0 qvarn @@ -1,8 +1,6 @@ #!/usr/bin/python3 # -# setup.py - standard Python build-and-package program -# -# Copyright 2017 Vincent Sanders <vince@qvarnlabs.com> +# Copyright 2017 Lars Wirzenius <liw@qvarnlabs.com> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Affero General Public License as published by @@ -20,14 +18,14 @@ from setuptools import setup -import qvarn +import salami setup( - name='qvarn-jsonb', - version=qvarn.__version__, - description='backend service for JSON and binary data storage', + name='salami', + version=salami.__version__, + description='thing', author='Lars Wirzenius', author_email='liw@qvarnlabs.com', - packages=['qvarn'], + packages=['salami'], ) diff --git a/start_debug_salami b/start_debug_salami new file mode 100755 index 0000000..7ca166b --- /dev/null +++ b/start_debug_salami @@ -0,0 +1,56 @@ +#!/bin/sh +# Copyright (C) 2017 Lars Wirzenius +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as +# published by the Free Software Foundation, either version 3 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +set -eu + +default_scopes=" +uapi_version_get +" + +tmp="$(mktemp -d)" + +cleanup() +{ + rm -rf "$tmp" +} + +trap cleanup EXIT + +token="$1" +case "$#" in + 2) key="$2" ;; + 1) key="$tmp/key" ;; +esac + + + +ISS=test +AUD=aud +./generate-rsa-key "$key" +./create-token "$key" "$ISS" "$AUD" "$default_scopes" > "$token" + +cat <<EOF > "$tmp/salami.yaml" +log: + - filename: salami.log +token-issuer: $ISS +token-audience: $AUD +token-public-key: $(cat "$key.pub") +EOF +export SALAMI_CONFIG="$tmp/salami.yaml" + +gunicorn3 --bind 127.0.0.1:12765 -p "$tmp/pid" -w1 --log-file g.log \ + --log-level debug \ + salami.backend:app diff --git a/start_qvarn b/start_salami index d446843..96dce90 100755 --- a/start_qvarn +++ b/start_salami @@ -16,11 +16,11 @@ set -eu -export QVARN_CONFIG="/etc/qvarn/qvarn.conf" +export START_CONFIG="/etc/salami/salami.yaml" gunicorn3 \ --bind 127.0.0.1:12765 \ -w1 \ --log-file /var/log/qvarn/gunicorn3.log \ --log-level debug \ - qvarn.backend:app + salami.backend:app diff --git a/without-tests b/without-tests index a752d9d..fac0e45 100644 --- a/without-tests +++ b/without-tests @@ -1,16 +1,9 @@ setup.py -qvarn/__init__.py -qvarn/api_errors.py -qvarn/backend.py -qvarn/file_router.py -qvarn/logging.py -qvarn/notification_router.py -qvarn/resource_router.py -qvarn/responses.py -qvarn/router.py -qvarn/sql.py -qvarn/subresource_router.py -qvarn/timestamp.py -qvarn/version.py -qvarn/version_router.py -yarns/lib.py +salami/__init__.py +salami/api.py +salami/backend.py +salami/log_setup.py +salami/responses.py +salami/router.py +salami/version.py +salami/version_router.py diff --git a/yarns/900-implements.yarn b/yarns/900-implements.yarn deleted file mode 100644 index 2f7ee0f..0000000 --- a/yarns/900-implements.yarn +++ /dev/null @@ -1,272 +0,0 @@ -# Step implementations - -This chapter shows the scenario step implementations. - -## Start and stop Qvarn - -Start a Qvarn running in the background. - - IMPLEMENTS GIVEN a running qvarn instance - import os, time, cliapp, yaml, yarnutils - privkey, pubkey = create_token_signing_key_pair() - print repr(privkey) - print repr(pubkey) - assert privkey - assert pubkey - open('key', 'w').write(privkey) - vars['aud'] = 'http://api.test.example.com' - vars['iss'] = 'qvarn.yarn' - vars['privkey'] = privkey - vars['pubkey'] = pubkey - vars['api.log'] = 'qvarn.log' - vars['gunicorn3.log'] = 'gunicorn3.log' - vars['pid-file'] = 'pid' - vars['port'] = cliapp.runcmd([os.path.join(srcdir, 'randport' )]).strip() - vars['url'] = 'http://127.0.0.1:{}'.format(vars['port']) - vars['API_URL'] = vars['url'] - config = { - 'log': [ - { - 'filename': vars['api.log'], - }, - ], - 'baseurl': vars['url'], - 'token-issuer': vars['iss'], - 'token-audience': vars['aud'], - 'token-public-key': vars['pubkey'], - 'resource-type-dir': os.path.join(srcdir, 'resource_type'), - } - config = add_postgres_config(config) - env = dict(os.environ) - env['QVARN_CONFIG'] = os.path.join(datadir, 'qvarn.yaml') - yaml.safe_dump(config, open(env['QVARN_CONFIG'], 'w')) - argv = [ - 'gunicorn3', - '--daemon', - '--bind', '127.0.0.1:{}'.format(vars['port']), - '-p', vars['pid-file'], - 'qvarn.backend:app', - ] - cliapp.runcmd(argv, env=env, stdout=None, stderr=None) - until = time.time() + 2.0 - while time.time() < until and not os.path.exists(vars['pid-file']): - time.sleep(0.01) - assert os.path.exists(vars['pid-file']) - -## Stop a Qvarn we started - - IMPLEMENTS FINALLY qvarn is stopped - import os, signal, yarnutils - pid = int(cat(vars['pid-file'])) - os.kill(pid, signal.SIGTERM) - -## API requests of various kinds - - IMPLEMENTS WHEN client requests GET (/.+) without token - path = get_next_match() - path = expand_vars(path, vars) - vars['status_code'], vars['headers'], vars['body'] = get(vars['url'] + path) - - IMPLEMENTS WHEN client requests GET (/.+) using token - path = get_next_match() - path = expand_vars(path, vars) - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - } - vars['status_code'], vars['headers'], vars['body'] = get( - vars['url'] + path, headers) - - IMPLEMENTS WHEN client requests POST (/.+) with token and body (.+) - path = get_next_match() - body = get_next_match() - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - 'Content-Type': 'application/json', - } - vars['status_code'], vars['headers'], vars['body'] = post( - vars['url'] + path, headers=headers, body=body) - - IMPLEMENTS WHEN client requests PUT (/.+) with token and body (.+) - path = get_next_match() - path = expand_vars(path, vars) - body = get_next_match() - body = expand_vars(body, vars) - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - 'Content-Type': 'application/json', - } - vars['status_code'], vars['headers'], vars['body'] = put( - vars['url'] + path, headers=headers, body=body) - - IMPLEMENTS WHEN client requests PUT (/[a-z0-9/${}]+) with token, revision (\S+), content-type (\S+), and empty body - path = expand_vars(get_next_match(), vars) - revision = expand_vars(get_next_match(), vars) - ctype = expand_vars(get_next_match(), vars) - body = '' - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - 'Revision': revision, - 'Content-Type': ctype, - } - vars['status_code'], vars['headers'], vars['body'] = put( - vars['url'] + path, headers=headers, body=body) - - IMPLEMENTS WHEN client requests PUT (/[a-z0-9/${}]+) with token, revision (\S+), content-type (\S+), and body "(.+)" - path = expand_vars(get_next_match(), vars) - revision = expand_vars(get_next_match(), vars) - ctype = expand_vars(get_next_match(), vars) - body = unescape(expand_vars(get_next_match(), vars)) - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - 'Revision': revision, - 'Content-Type': ctype, - } - vars['status_code'], vars['headers'], vars['body'] = put( - vars['url'] + path, headers=headers, body=body) - - IMPLEMENTS WHEN client requests DELETE (/.+) with token - path = get_next_match() - path = expand_vars(path, vars) - headers = { - 'Authorization': 'Bearer {}'.format(vars['token']), - } - vars['status_code'], vars['headers'], vars['body'] = delete( - vars['url'] + path, headers=headers) - - IMPLEMENTS WHEN client uploads a fake jpg - assert 0 - -## API access token creation - - IMPLEMENTS WHEN client gets an authorization token with scope "(.+)" - scopes = get_next_match() - print 'privkey', repr(vars['privkey']) - assert vars['privkey'] - vars['token'] = create_token(vars['privkey'], vars['iss'], vars['aud'], scopes) - -## UUID creation - - IMPLEMENTS GIVEN unique random identifier (\S+) - import uuid - name = get_next_match() - vars[name] = str(uuid.uuid4()) - -## API request result checking - - IMPLEMENTS THEN HTTP status code is (\d+) (.*) - expected = int(get_next_match()) - assertEqual(vars['status_code'], expected) - - IMPLEMENTS THEN HTTP (\S+) header is (.+) - header = get_next_match() - value = expand_vars(get_next_match(), vars) - assertEqual(vars['headers'].get(header), value) - - IMPLEMENTS THEN remember HTTP (\S+) header as (.+) - header = get_next_match() - name = get_next_match() - vars[name] = vars['headers'].get(header) - - IMPLEMENTS THEN resource id is (\S+) - import json - name = get_next_match() - print 'body:', repr(vars['body']) - body = json.loads(vars['body']) - vars[name] = body['id'] - - IMPLEMENTS THEN revision is (\S+) - import json - name = get_next_match() - body = json.loads(vars['body']) - vars[name] = body['revision'] - - IMPLEMENTS THEN revisions (\S+) and (\S+) are different - rev1 = get_next_match() - rev2 = get_next_match() - assertNotEqual(vars[rev1], vars[rev2]) - - IMPLEMENTS THEN revisions (\S+) and (\S+) match - rev1 = get_next_match() - rev2 = get_next_match() - assertEqual(vars[rev1], vars[rev2]) - - IMPLEMENTS THEN JSON body matches (.+) - import json - wanted = get_next_match() - print 'wanted1', repr(wanted) - wanted = expand_vars(wanted, vars) - print 'wanted2', repr(wanted) - wanted = json.loads(wanted) - actual = json.loads(vars['body']) - print 'actual ', repr(actual) - print 'wanted3', repr(wanted) - assertTrue(values_match(wanted, actual)) - - IMPLEMENTS THEN body is "(.+)" - wanted = unescape(expand_vars(get_next_match(), vars)) - body = vars['body'] - assertTrue(values_match(wanted, body)) - - IMPLEMENTS THEN search result contains (.+) - import json - wanted1 = get_next_match() - wanted2 = expand_vars(wanted1, vars) - wanted = json.loads(wanted2) - actual = json.loads(vars['body']) - print 'wanted1:', repr(wanted1) - print 'wanted2:', repr(wanted2) - print 'wanted:', repr(wanted) - print 'actual:', repr(actual) - assertTrue(actual['resources']) - found = False - for result in actual['resources']: - if values_match(wanted, result): - print 'MATCH!', repr(wanted), repr(result) - found = True - break - print 'no match', repr(wanted), repr(result) - assertTrue(found) - - IMPLEMENTS THEN search result does NOT contain (.+) - import json - wanted1 = get_next_match() - wanted2 = expand_vars(wanted1, vars) - wanted = json.loads(wanted2) - actual = json.loads(vars['body']) - print 'wanted1:', repr(wanted1) - print 'wanted2:', repr(wanted2) - print 'wanted:', repr(wanted) - print 'actual:', repr(actual) - found = False - for result in actual['resources']: - if values_match(wanted, result): - found = True - assertFalse(found) - - IMPLEMENTS THEN search result at index (\d+) has id (\S+) - import json - index = int(get_next_match()) - id_name = get_next_match() - body = json.loads(vars['body']) - print 'body', repr(body) - resources = body['resources'] - print 'resources', repr(resources) - print 'len resources', len(resources) - print 'index', index - assert index < len(resources) - obj = resources[index] - print 'resource at index', repr(obj) - print 'id', repr(obj['id']) - vars[id_name] = obj['id'] - - IMPLEMENTS THEN search result has (\d+) resources - wanted = int(get_next_match()) - body = json.loads(vars['body']) - print 'body', repr(body) - resources = body['resources'] - print 'resources', repr(resources) - print 'len resources', len(resources) - assertEqual(wanted, len(resources)) - - IMPLEMENTS THEN response has header WWW-Authenticate containing "(.+)" - assert 0 diff --git a/yarns/lib.py b/yarns/lib.py deleted file mode 100644 index 459a821..0000000 --- a/yarns/lib.py +++ /dev/null @@ -1,157 +0,0 @@ -# Copyright (C) 2017 Lars Wirzenius -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as -# published by the Free Software Foundation, either version 3 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with this program. If not, see <http://www.gnu.org/licenses/>. - - -import json -import os -import re -import tempfile - - -import cliapp -import Crypto.PublicKey.RSA -import requests -import yaml - - -from yarnutils import * - - -srcdir = os.environ['SRCDIR'] -datadir = os.environ['DATADIR'] - - -vars = Variables(datadir) - - -def hexdigit(c): - return ord(c) - ord('0') - - -def unescape(s): - t = '' - while s: - if s.startswith('\\x') and len(s) >= 4: - a = hexdigit(s[2]) - b = hexdigit(s[3]) - t += chr(a * 16 + b) - s = s[4:] - else: - t += s[0] - s = s[1:] - return t - - -def add_postgres_config(config): - pg = os.environ.get('QVARN_POSTGRES') - if pg: - with open(pg) as f: - config['database'] = yaml.safe_load(f) - config['memory-database'] = False - return config - - -def get(url, headers=None): - print('get: url={} headers={}'.format(url, headers)) - r = requests.get(url, headers=headers) - return r.status_code, dict(r.headers), r.content - - -def post(url, headers=None, body=None): - r = requests.post(url, headers=headers, data=body) - return r.status_code, dict(r.headers), r.text - - -def put(url, headers=None, body=None): - r = requests.put(url, headers=headers, data=body) - return r.status_code, dict(r.headers), r.text - - -def delete(url, headers=None): - r = requests.delete(url, headers=headers) - return r.status_code, dict(r.headers), r.text - - -def create_token_signing_key_pair(): - RSA_KEY_BITS = 4096 # A nice, currently safe length - key = Crypto.PublicKey.RSA.generate(RSA_KEY_BITS) - return key.exportKey('PEM'), key.exportKey('OpenSSH') - - -def create_token(privkey, iss, aud, scopes): - filename = write_temp(privkey) - argv = [ - os.path.join(srcdir, 'create-token'), - filename, - iss, - aud, - scopes, - ] - return cliapp.runcmd(argv) - - -def cat(filename): - return open(filename).read() - - -def write_temp(data): - fd, filename = tempfile.mkstemp(dir=datadir) - os.write(fd, data) - os.close(fd) - return filename - - -def expand_vars(text, vars): - result = '' - while text: - m = re.search(r'\${(?P<name>[^}]+)}', text) - if not m: - result += text - break - name = m.group('name') - print('expanding ', name) - result += text[:m.start()] + vars[name] - text = text[m.end():] - return result - - -def values_match(wanted, actual): - print - print 'wanted:', repr(wanted) - print 'actual:', repr(actual) - - if type(wanted) != type(actual): - print 'wanted and actual types differ', type(wanted), type(actual) - return False - - if isinstance(wanted, dict): - for key in wanted: - if key not in actual: - print 'key {!r} not in actual'.format(key) - return False - if not values_match(wanted[key], actual[key]): - return False - elif isinstance(wanted, list): - if len(wanted) != len(actual): - print 'wanted and actual are of different lengths' - for witem, aitem in zip(wanted, actual): - if not values_match(witem, aitem): - return False - else: - if wanted != actual: - print 'wanted and actual differ' - return False - - return True diff --git a/yarns/smoke.yarn b/yarns/smoke.yarn deleted file mode 100644 index 6e774d2..0000000 --- a/yarns/smoke.yarn +++ /dev/null @@ -1,1204 +0,0 @@ ---- -title: qvarn-jsonb integration tests -author: Lars Wirzenius / QvarnLabs Ab -date: work in progress -... - - -# Introduction - -This is an integration test suite for the Qvarn HTTP API. - -## History and background - -Qvarn started as an internal project in 2015 at Suomen Tilaajavastuu -(https://www.tilaajavastuu.fi/en/), a company that provides various -reproting services to the Finnish construction industry. Later that -year the Swedish Construction Federation -(https://www.sverigesbyggindustrier.se/english) adopted Qvarn as a -platform for ID06 identity card services in Sweden. From the start, a -goal for Qvarn has been to make it easy to comply with the EU General -Data Protection Regulation. - -In 2016 Qvarn spun off to its own company, QvarnLabs Ab, which -provides consulting, development, and support services around Qvarn. - - -# Version checking - - SCENARIO Qvarn reports its version - - GIVEN a running qvarn instance - - WHEN client requests GET /version without token - THEN HTTP status code is 200 OK - - WHEN client gets an authorization token with scope "uapi_version_get" - AND client requests GET /version using token - THEN HTTP status code is 200 OK - - FINALLY qvarn is stopped - - -# Manage a subject - - SCENARIO user manages a subject resource - - GIVEN a running Qvarn instance - - WHEN client requests GET /subjects without token - THEN HTTP status code is 401 Unauthorized - - WHEN client requests GET /subjects/notexist without token - THEN HTTP status code is 401 Unauthorized - - WHEN client gets an authorization token with scope - ... "uapi_subjects_get uapi_subjects_post uapi_subjects_id_get - ... uapi_subjects_id_put uapi_subjects_id_delete" - AND client requests GET /subjects using token - THEN HTTP status code is 200 OK - - WHEN client requests GET /subjects using token - THEN HTTP status code is 200 OK - AND search result does NOT contain { "id": "subject" } - - WHEN client requests POST /subjects with token and body - ... { - ... "type": "subject", - ... "names": [ - ... { "full_name": "Jason Bourne" } - ... ] - ... } - THEN HTTP status code is 201 Created - AND resource id is ID1 - AND revision is REV1 - - WHEN client requests GET /subjects/${ID1} without token - THEN HTTP status code is 401 Unauthorized - - WHEN client requests GET /subjects/${ID1} using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { "revision": "${REV1}", "id": "${ID1}", "type": "subject", - ... "names": [{ "full_name": "Jason Bourne" }] } - -For silly hysterical raisins, the Qvarn API is defined to return 400 -if updating a resource with the wrong revision in the body, instead -of 404. We may want to fix this some day. - - WHEN client requests PUT /subjects/${ID1} with token and body - ... { "id": "wrong", "revision": "${REV1}", "names": [{"full_name": "Dave Webb"}]} - THEN HTTP status code is 400 Bad - - WHEN client requests PUT /subjects/${ID1} with token and body - ... { "id": "${ID1}", "revision": "${REV1}", "names": [{"full_name": "Dave Webb"}]} - THEN HTTP status code is 200 OK - AND revision is REV2 - - WHEN client requests PUT /subjects/${ID1} with token and body - ... { "type": "subject", "id": "${ID1}", "revision": "${REV2}", - ... "names": [{"full_name": "David Webb" }]} - THEN HTTP status code is 200 OK - AND revision is REV3 - - WHEN client requests GET /subjects/${ID1} using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { "revision": "${REV3}", "id": "${ID1}", "type": "subject", - ... "names": [{"full_name": "David Webb" }]} - - WHEN client requests DELETE /subjects/${ID1} with token - THEN HTTP status code is 200 OK - - WHEN client requests GET /subjects/${ID1} using token - THEN HTTP status code is 404 Not Found - - FINALLY qvarn is stopped - - -# Manage notifications - -Notifications are a special resource type. Notifications created -automatically by Qvarn, but the API client may delete them. - - SCENARIO manage notifications - - GIVEN a running Qvarn instance - -Client has needed access rights for orgs resource. - - WHEN client gets an authorization token with scope - ... "uapi_orgs_listeners_post uapi_orgs_listeners_id_get - ... uapi_orgs_listeners_get uapi_orgs_listeners_id_notifications_get - ... uapi_orgs_post uapi_orgs_listeners_id_notifications_id_get - ... uapi_orgs_listeners_id_put uapi_orgs_id_put uapi_orgs_id_delete - ... uapi_orgs_listeners_id_delete - ... uapi_orgs_listeners_id_notifications_id_delete" - - WHEN client requests POST /orgs/listeners with token and body - ... { - ... "notify_of_new": true - ... } - THEN HTTP status code is 201 Created - AND JSON body matches - ... { - ... "type": "listener", - ... "notify_of_new": true, - ... "listen_on": [] - ... } - AND resource id is LISTENID1 - AND HTTP Location header is ${API_URL}/orgs/listeners/${LISTENID1} - - WHEN client requests POST /orgs/listeners with token and body - ... { - ... "notify_of_new": false - ... } - THEN HTTP status code is 201 Created - AND JSON body matches - ... { - ... "type": "listener", - ... "notify_of_new": false, - ... "listen_on": [] - ... } - AND resource id is LISTENID2 - AND HTTP Location header is ${API_URL}/orgs/listeners/${LISTENID2} - AND revision is REV1 - - WHEN client requests POST /orgs/listeners with token and body - ... { - ... "notify_of_new": false, - ... "listen_on_all": true - ... } - THEN HTTP status code is 201 Created - AND JSON body matches - ... { - ... "type": "listener", - ... "notify_of_new": false, - ... "listen_on_all": true, - ... "listen_on": [] - ... } - AND resource id is LISTENID3 - AND HTTP Location header is ${API_URL}/orgs/listeners/${LISTENID3} - - WHEN client requests GET /orgs/listeners/${LISTENID1} using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "id": "${LISTENID1}", - ... "type": "listener", - ... "notify_of_new": true, - ... "listen_on": [] - ... } - - WHEN client requests GET /orgs/listeners using token - THEN HTTP status code is 200 OK - THEN search result contains {"id": "${LISTENID1}"} - THEN search result contains {"id": "${LISTENID2}"} - -A listener has no notifications initially. - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [] - ... } - - WHEN client requests POST /orgs with token and body - ... { - ... "names": ["Universal Exports"] - ... } - THEN resource id is ORGID1 - AND revision is REV2 - - WHEN client requests POST /orgs with token and body - ... { - ... "names": ["Telebulvania Ltd"] - ... } - THEN resource id is ORGID2 - -After adding the new organizations the first listener should be notified while -the second and third should have no notifications. - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications - ... using token - THEN HTTP status code is 200 OK - AND search result at index 0 has id MSGID1 - AND search result at index 1 has id MSGID2 - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications/${MSGID1} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID1}", - ... "type": "notification", - ... "resource_id": "${ORGID1}", - ... "resource_change": "created" - ... } - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications/${MSGID2} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID2}", - ... "type": "notification", - ... "resource_id": "${ORGID2}", - ... "resource_change": "created" - ... } - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [] - ... } - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [] - ... } - -We update the empty listener to listen on organization changes and update the -organization checking for the correct notification to appear. The third -listener listening to all the changes should get the notification, too. - - WHEN client requests PUT /orgs/listeners/${LISTENID2} with token and body - ... { - ... "notify_of_new": false, - ... "listen_on": ["${ORGID1}"], - ... "revision": "${REV1}" - ... } - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "type": "listener", - ... "notify_of_new": false, - ... "listen_on": ["${ORGID1}"] - ... } - - WHEN client requests PUT /orgs/${ORGID1} with token and body - ... { - ... "names": ["Universal Experts"], - ... "revision": "${REV2}" - ... } - THEN HTTP status code is 200 OK - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications - ... using token - THEN HTTP status code is 200 OK - AND search result at index 0 has id MSGID3 - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications/${MSGID3} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID3}", - ... "type": "notification", - ... "resource_id": "${ORGID1}", - ... "resource_change": "updated" - ... } - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications - ... using token - THEN HTTP status code is 200 OK - AND search result has 1 resources - AND search result at index 0 has id MSGID4 - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications/${MSGID4} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID4}", - ... "type": "notification", - ... "resource_id": "${ORGID1}", - ... "resource_change": "updated" - ... } - -The first listener gets no additional notifications. - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${MSGID1}"}, {"id": "${MSGID2}"} - ... ] - ... } - -We delete the organization and check for the correct notification to appear. - - WHEN client requests DELETE /orgs/${ORGID1} with token - THEN HTTP status code is 200 OK - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications - ... using token - THEN HTTP status code is 200 OK - AND search result at index 1 has id MSGID5 - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications/${MSGID5} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID5}", - ... "type": "notification", - ... "resource_id": "${ORGID1}", - ... "resource_revision": null, - ... "resource_change": "deleted" - ... } - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications - ... using token - THEN HTTP status code is 200 OK - AND search result at index 1 has id MSGID6 - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications/${MSGID6} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID6}", - ... "type": "notification", - ... "resource_id": "${ORGID1}", - ... "resource_revision": null, - ... "resource_change": "deleted" - ... } - -The first listener gets no additional notifications. - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${MSGID1}"}, {"id": "${MSGID2}"} - ... ] - ... } - -Deletion of a listener deletes also the notifications. - - WHEN client requests DELETE /orgs/listeners/${LISTENID1} with token - THEN HTTP status code is 200 OK - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications/${MSGID1} - ... using token - THEN HTTP status code is 404 Not Found - - WHEN client requests - ... GET /orgs/listeners/${LISTENID1}/notifications/${MSGID2} - ... using token - THEN HTTP status code is 404 Not Found - - WHEN client requests GET /orgs/listeners/${LISTENID1} using token - THEN HTTP status code is 404 Not Found - -Notification can be deleted. - - WHEN client requests - ... DELETE /orgs/listeners/${LISTENID2}/notifications/${MSGID3} - ... with token - THEN HTTP status code is 200 OK - - WHEN client requests - ... GET /orgs/listeners/${LISTENID2}/notifications/${MSGID3} - ... using token - THEN HTTP status code is 404 Not Found - - WHEN client requests - ... DELETE /orgs/listeners/${LISTENID3}/notifications/${MSGID4} - ... with token - THEN HTTP status code is 200 OK - - WHEN client requests - ... GET /orgs/listeners/${LISTENID3}/notifications/${MSGID4} - ... using token - THEN HTTP status code is 404 Not Found - - WHEN client requests - ... DELETE /orgs/listeners/${LISTENID2}/notifications/${MSGID5} - ... with token - THEN HTTP status code is 200 OK - - WHEN client requests - ... DELETE /orgs/listeners/${LISTENID3}/notifications/${MSGID6} - ... with token - THEN HTTP status code is 200 OK - - - FINALLY qvarn is stopped - -# Listeners for one resource type only - -This scenario tests for a problem found in the first public release of -Qvarn (JSONB): all listeners would be reported for all resource types, -intead of only the one they were created for. - - SCENARIO notifications only for the right type - - GIVEN a running qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_persons_post - ... uapi_persons_listeners_post - ... uapi_persons_listeners_get - ... uapi_persons_listeners_id_notifications_get - ... uapi_persons_listeners_id_notifications_id_get - ... uapi_orgs_post - ... uapi_orgs_listeners_post - ... uapi_orgs_listeners_get - ... uapi_orgs_listeners_id_notifications_get - ... uapi_orgs_listeners_id_notifications_id_get - ... " - -Make sure there are no listeners yet. - - WHEN client requests GET /orgs/listeners using token - THEN JSON body matches { "resources": [] } - - WHEN client requests GET /persons/listeners using token - THEN JSON body matches { "resources": [] } - -Create a listeners for orgs. - - WHEN client requests POST /orgs/listeners with token and body - ... { - ... "notify_of_new": true - ... } - THEN resource id is ORGLISTENER - - WHEN client requests GET /orgs/listeners using token - THEN JSON body matches { "resources": [{ "id": "${ORGLISTENER}" }]} - - WHEN client requests GET /persons/listeners using token - THEN JSON body matches { "resources": [] } - -Create a listener for persons. - - WHEN client requests POST /persons/listeners with token and body - ... { - ... "notify_of_new": true - ... } - THEN resource id is PERSONLISTENER - - WHEN client requests GET /orgs/listeners using token - THEN JSON body matches { "resources": [{ "id": "${ORGLISTENER}" }]} - - WHEN client requests GET /persons/listeners using token - THEN JSON body matches { "resources": [{ "id": "${PERSONLISTENER}"}]} - -Create a person, make sure only the correct notifications are created. - - WHEN client requests POST /persons with token and body - ... { - ... "names": [{ "full_name": "James Bond" }] - ... } - THEN resource id is PERSONID - - WHEN client requests - ... GET /orgs/listeners/${ORGLISTENER}/notifications - ... using token - THEN JSON body matches { "resources": [] } - - WHEN client requests - ... GET /persons/listeners/${PERSONLISTENER}/notifications - ... using token - THEN search result at index 0 has id MSGID - - WHEN client requests - ... GET /persons/listeners/${PERSONLISTENER}/notifications/${MSGID} - ... using token - THEN JSON body matches - ... { - ... "id": "${MSGID}", - ... "type": "notification", - ... "resource_id": "${PERSONID}", - ... "resource_change": "created" - ... } - - FINALLY qvarn is stopped - -# Use subresources - -Subresources are additional resources that are attached to a resource, -and be managed and access controlled separetly. However, they share -existence and revision with their parent: if the parent is deleted, so -is the subresource, and if either the parent or the subresource is -changed, and gets a new revision, so does the other. - -Subresources always exist, if defined in the resource type spec. They -don't need to be created specially. - - SCENARIO manage subresources - - GIVEN a running qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_sub_put uapi_subjects_sub_get" - - WHEN client requests POST /subjects with token and body - ... { "type": "subject", "names": [ { "full_name": "Alfred" } ] } - THEN resource id is ID - AND revision is REV - - WHEN client requests GET /subjects/${ID}/sub using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "subfield": "" - ... } - - WHEN client requests PUT /subjects/${ID}/sub with token and body - ... { - ... "revision": "wrong", - ... "subfield": "Steven Segal" - ... } - THEN HTTP status code is 409 Conflict - - WHEN client requests PUT /subjects/${ID}/sub with token and body - ... { - ... "revision": "${REV}", - ... "subfield": "Steven Segal" - ... } - THEN HTTP status code is 200 OK - AND revision is REV2 - AND JSON body matches - ... { - ... "revision": "${REV2}", - ... "subfield": "Steven Segal" - ... } - - FINALLY qvarn is stopped - - -# Manage blobs - -Blobs are like sub-resources, but they're arbitrary binary data, not -JSON. - - SCENARIO manage blobs - - GIVEN a running qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_blob_put uapi_subjects_blog_get" - -Create a subject. - - WHEN client requests POST /subjects with token and body - ... { "type": "subject", "names": [ { "full_name": "Alfred" } ] } - THEN resource id is ID - AND revision is REV1 - -Newly created subject does not have a blob. - - WHEN client requests GET /subjects/${ID}/blob using token - THEN HTTP status code is 404 Not found - -Uploading an empty blob doesn't work. - - WHEN client requests PUT /subjects/${ID}/blob with token, - ... revision REV1, - ... content-type image/jpeg, - ... and empty body - THEN HTTP status code is 411 Length required - -Uploading a COMPLETELY VALID JPEG as a blob fails, if subject resource -revision is wrong. - - WHEN client requests PUT /subjects/${ID}/blob with token, - ... revision BADREV, - ... content-type image/jpeg, - ... and body "FAKE JPEG" - THEN HTTP status code is 409 Conflict - -Uploading with valid revision works. - - WHEN client requests PUT /subjects/${ID}/blob with token, - ... revision ${REV1}, - ... content-type image/jpeg, - ... and body "FAKE JPEG" - THEN HTTP status code is 200 OK - -Do we get the right blob back? Also, note that subject revision -should've changed. - - WHEN client requests GET /subjects/${ID}/blob using token - THEN HTTP status code is 200 OK - AND HTTP Content-Type header is image/jpeg - AND body is "FAKE JPEG" - AND remember HTTP Revision header as REV2 - AND revisions REV1 and REV2 are different - -Uploading with old revision fails. - - WHEN client requests PUT /subjects/${ID}/blob with token, - ... revision ${REV1}, - ... content-type image/jpeg, - ... and body "FAKE JPEG" - THEN HTTP status code is 409 Conflict - -Uploading a new blob with the current revision works. - - WHEN client requests PUT /subjects/${ID}/blob with token, - ... revision ${REV2}, - ... content-type image/jpeg, - ... and body "\x89" - THEN HTTP status code is 200 OK - AND remember HTTP Revision header as REV3 - -And it did get updated. - - WHEN client requests GET /subjects/${ID}/blob using token - THEN HTTP status code is 200 OK - AND HTTP Content-Type header is image/jpeg - AND body is "\x89" - -Updating parent doesn't affect the blob. - - WHEN client requests PUT /subjects/${ID} with token and body - ... { - ... "type": "subject", - ... "revision": "${REV3}", - ... "names": [ { "full_name": "Melissa" } ] - ... } - THEN revision is REV4 - - WHEN client requests GET /subjects/${ID}/blob using token - THEN HTTP status code is 200 OK - AND HTTP Content-Type header is image/jpeg - AND body is "\x89" - AND remember HTTP Revision header as REV5 - AND revisions REV4 and REV5 match - - FINALLY qvarn is stopped - -# Search subjects - - SCENARIO search subjects - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_search_id_get - ... uapi_subjects_id_delete" - - WHEN client requests POST /subjects with token and body - ... { "type": "subject", "names": [ { "full_name": "Alfred" } ] } - THEN resource id is ID1 - AND revision is REV1 - - WHEN client requests POST /subjects with token and body - ... { "type": "subject", "names": [ { "full_name": "Alfred" } ] } - THEN resource id is ID2 - - WHEN client requests POST /subjects with token and body - ... { "type": "subject", "names": [ { "full_name": "Bruce" } ] } - THEN resource id is ID3 - - WHEN client requests GET /subjects/search/exact/full_name/Batman - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches { "resources": []} - -Make sure searches are case-insensitive. - - WHEN client requests GET /subjects/search/exact/full_name/ALFRED - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result does NOT contain {"id": "${ID3}"} - - WHEN client requests GET /subjects/search/exact/full_name/bruce - ... using token - THEN HTTP status code is 200 OK - AND search result does NOT contain {"id": "${ID1}"} - AND search result does NOT contain {"id": "${ID2}"} - AND search result contains {"id": "${ID3}"} - AND JSON body matches { "resources": [{"id": "${ID3}"}]} - - WHEN client requests - ... GET /subjects/search/contains/full_name/fred - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result does NOT contain {"id": "${ID3}"} - - WHEN client requests - ... GET /subjects/search/gt/full_name/Alfred - ... using token - THEN HTTP status code is 200 OK - AND search result does NOT contain {"id": "${ID1}"} - AND search result does NOT contain {"id": "${ID2}"} - AND search result contains {"id": "${ID3}"} - - WHEN client requests - ... GET /subjects/search/ge/full_name/Alfred - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result contains {"id": "${ID3}"} - - WHEN client requests - ... GET /subjects/search/startswith/full_name/Alfred - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result does NOT contain {"id": "${ID3}"} - - WHEN client requests - ... GET /subjects/search/show/full_name/contains/full_name/fred - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result does NOT contain {"id": "${ID3}"} - - WHEN client requests - ... GET /subjects/search/show_all/exact/full_name/Bruce - ... using token - THEN HTTP status code is 200 OK - AND search result contains - ... { - ... "id": "${ID3}", - ... "names": [ { "full_name": "Bruce" } ] - ... } - - WHEN client requests - ... GET /subjects/search/exact/full_name/Br%2Fce - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches { "resources": [] } - - GIVEN unique random identifier RANDOM - WHEN client requests PUT /subjects/${ID1}/sub with token and body - ... { "subfield": "${RANDOM}", "revision": "${REV1}" } - THEN HTTP status code is 200 OK - - WHEN client requests GET /subjects/search/exact/subfield/${RANDOM} using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - - FINALLY qvarn is stopped - - -# More search - -This scenario creates two resources, and searches with two conditions. -Each condition matches a resource, but the two match different ones. -Thus the result should be an empty set. - - SCENARIO search with two conditions when two resources match one - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_search_id_get - ... uapi_subjects_id_delete" - - WHEN client requests POST /subjects with token and body - ... { - ... "type": "subject", - ... "names": [{ - ... "full_name": "Clark Kent", - ... "sort_key": "Clark the superman" - ... }] - ... } - THEN resource id is ID1 - - WHEN client requests POST /subjects with token and body - ... { - ... "type": "subject", - ... "names": [{ - ... "full_name": "Clark Celt", - ... "sort_key": "a nobody" - ... }] - ... } - THEN resource id is ID2 - - WHEN client requests - ... GET /subjects/search/contains/full_name/Kent/contains/sort_key/nobody - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches { "resources": [ ]} - - WHEN client requests - ... GET /subjects/search/contains/full_name/Clark - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... {"resources": [{"id": "${ID1}"}, {"id": "${ID2}"}]} - - WHEN client requests - ... GET /subjects/search/contains/full_name/Kent - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches {"resources": [{"id": "${ID1}"}]} - - WHEN client requests - ... GET /subjects/search/contains/sort_key/nobody - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches {"resources": [{"id": "${ID2}"}]} - - FINALLY qvarn is stopped - -# Searching with multiple conditions - - SCENARIO search with multiple conditions - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_contracts_post uapi_contracts_search_id_get" - -Create an organisation, a person and an employment contract between them. - - WHEN client requests POST /contracts with token and body - ... {"contract_type": "employment", - ... "start_date": "2016-01-01", - ... "contract_parties": [ - ... { - ... "type": "org", - ... "resource_id": "org-1", - ... "role": "employer" - ... }, - ... { - ... "type": "person", - ... "resource_id": "person-1", - ... "role": "employee" - ... } - ... ], - ... "contract_state": "active" - ... } - THEN HTTP status code is 201 Created - AND resource id is CONTRACT_ID1 - -Perform searches matching different instances of the same nested element. - - WHEN client requests - ... GET /contracts/search/exact/resource_id/org-1/exact/resource_id/person-1 - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${CONTRACT_ID1}"} - - WHEN client requests - ... GET /contracts/search/exact/resource_id/org-1/exact/resource_id/wrong_id - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [] - ... } - - FINALLY qvarn is stopped - -# Sort search results - - SCENARIO search with sort - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_search_id_get" - -Create several person resources. - - GIVEN unique random identifier UID - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person A", - ... "sort_key": "3, Person", - ... "given_names": ["Test"], - ... "surnames": ["Person", "A"] - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID3 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person B", - ... "sort_key": "1, Person", - ... "given_names": ["Test"], - ... "surnames": ["Person", "B"] - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID1 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person C", - ... "sort_key": "2, Person", - ... "given_names": ["Test"], - ... "surnames": ["Person", "C"] - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID2 - -Search person resources and sort results by `sort_key`. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/sort_key - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID1}"}, - ... {"id": "${ID2}"}, - ... {"id": "${ID3}"} - ... ] - ... } - -Sort person resources using different sort key. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/full_name - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID3}"}, - ... {"id": "${ID1}"}, - ... {"id": "${ID2}"} - ... ] - ... } - -Search person resources and sort results by two search keys, where first search -key is a list containing more than one value. First key is `surnames`, where -each resource has same first surname, and second key is `sort_key`. Since each -first surname is the same, results should fall back to the second sort key. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/surnames/sort/sort_key - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID1}"}, - ... {"id": "${ID2}"}, - ... {"id": "${ID3}"} - ... ] - ... } - -Search with only search operator should also work, returning all available -resource ids. - - WHEN client requests - ... GET /subjects/search/sort/sort_key - ... using token - THEN HTTP status code is 200 OK - AND search result contains {"id": "${ID1}"} - AND search result contains {"id": "${ID2}"} - AND search result contains {"id": "${ID3}"} - - FINALLY qvarn is stopped - -# Sort search results with /offset and /limit - - SCENARIO search with /offset and /limit - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_subjects_post uapi_subjects_search_id_get" - -Create several person resources. - - GIVEN unique random identifier UID - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person 1" - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID1 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person 2" - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID2 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person 3" - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID3 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person 4" - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID4 - - WHEN client requests POST /subjects with token and body - ... { - ... "random_id": "${UID}", - ... "names": [ - ... { - ... "full_name": "Person 5" - ... } - ... ] - ... } - THEN HTTP status code is 201 Created - THEN resource id is ID5 - -Sort, return first two hits. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/full_name/limit/2 - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID1}"}, - ... {"id": "${ID2}"} - ... ] - ... } - -Sort, return second set of two hits. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/full_name/offset/2/limit/2 - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID3}"}, - ... {"id": "${ID4}"} - ... ] - ... } - -Sort, return third set of two hits, which is actualy only one item. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/sort/full_name/offset/4/limit/2 - ... using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "resources": [ - ... {"id": "${ID5}"} - ... ] - ... } - -Don't sort. Then /offset and /limit are verboten. - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/offset/1 - ... using token - THEN HTTP status code is 400 Error - AND JSON body matches - ... { - ... "message": "LIMIT and OFFSET can only be used with together SORT.", - ... "error_code": "LimitWithoutSortError" - ... } - - WHEN client requests - ... GET /subjects/search/exact/random_id/${UID}/offset/1 - ... using token - THEN HTTP status code is 400 Error - AND JSON body matches - ... { - ... "message": "LIMIT and OFFSET can only be used with together SORT.", - ... "error_code": "LimitWithoutSortError" - ... } - - FINALLY qvarn is stopped - - -# Handle resource types via API - -Qvarn API allows listing and looking at all the resource types it -knows about. In the future, it will allow manipulaing them as well. - - SCENARIO manage resource types - - GIVEN a running Qvarn instance - - WHEN client gets an authorization token with scope - ... "uapi_resource_types_get uapi_resource_types_id_get" - - WHEN client requests GET /resource_types using token - THEN HTTP status code is 200 OK - AND search result contains { "id": "subject" } - - WHEN client requests GET /resource_types/subject using token - THEN HTTP status code is 200 OK - AND JSON body matches - ... { - ... "id": "subject", - ... "type": "resource_type", - ... "path": "/subjects" - ... } - - FINALLY qvarn is stopped |