EuroPython 2014

Lightning Talks [EuroPython 2014]

Welcome [EuroPython 2014] [21 July 2014]

pymove3D Winner Announcement [EuroPython 2014[ [21 July 2014]

Daniel Kraft/Oliver Roch - Using python, LXC and linux to create a mass VM hosting, managed by django and angularjs [EuroPython 2014] [22 July 2014] How we created a scalable mass VM hosting for open source web apps with python, LXC and linux with a web-UI based on django and angularjs. We'll show the underlying architecture of this service, several linux internals that make this possible and we'll talk about bitter failure during development. This talk will be python- and linux-centric with some hints for integrating angularjs into django.

Schlomo Shapiro - Sponsoring Open Source [EuroPython 2014] und damit den Chef überzeugen

Erik Janssens - SQLAlchemy Drill [EuroPython 2014] [23 July 2014] If you have been looking to use SQLAlchemy in one of your projects, but found the documentation a bit overwhelming then this talk is for you. If you have used SQLAlchemy but feel there are some holes in your knowledge of the library, then this talk is for you as well. The idea is that during this talk you bring your laptop with you, and make sure you have SQLAlchemy installed. At the beginning of the talk, we fire up our Python interpreter and start to explore the library in a structured way. In the next 25 minutes, we'll go hands on through the various parts of the SQLAlchemy. We try out the concepts of each part of the library and make sure the basics are well understood. ----- In this talk will introduce the audience to SQLAlchemy in a well structured way, so that basic concepts are understood. This talk will be a combination of slides and interactive code editing in IPython. Both the working of SQLAlchemy as well as best practices in using SQLAlchemy will be demonstrated. I will demonstrate the basic workings of: * the SQL generation layer * the DDL generation * the ORM * the session * transactions The used code will allow those who have their laptop with them to try the code samples for themselves.

Thomas Aglassinger - Solution oriented error handling [EuroPython 2014] [22 July 2014] This talk shows how to use Python's built in error handling mechanisms to keep the productive code clean, derive error messages helpful for the user directly from the code and release ressources properly. ----- Traditionally error handling is regarded an annoyance by developers because it removes the focus from the already difficult enough productive parts of the code to parts that ideally will never be called. And even if, end users seem to be ignore the error messages and just click "Ok" or call the help desk. Solution oriented error handling uses Python's existing try/catch/finally idiom, with statement, assert statement and exception hierarchy in a way that keeps the code clean and easy to maintain. It gives a clear distinction between errors that can be solved by the end user, the system administrator and the developer. Naming conventions and a simple set of coding guidelines ensure that helpful error messages can be easily derived from the code. Most code examples work with Python 2.6+ and Python 3.x, on a few occasions minor differences are pointed out. Topics covered are: 1. Introduction to error handling in Python - What are errors? - How to represent errors in Python - Detecting errors - Delegating errors to the caller - clean resource management 2. Principles of solution oriented error handling - responsibilities between user, admin and developer - when to use raise or assert 3. Error messages - What are "good" error messages - How to derive error messages from the source code - Adding context to the error - How to report errors to the user 4. Solution oriented usage of Python's exception hierarchy - admins fix `EnvironmentError` - users fix `DataError` - representing `DataError` - converting exceptions to `DataError` - developers fix everything else - special Python exceptions not representing errors 5. Template for a solution oriented command line application 6. Best practices for `raise` and `except` - When to use `raise`? - When to use `except`? This talk is a translation of a German [talk](https://github.com/roskakori/talks/tree/master/pygraz/errorhandling) given at the PyGRAZ user group and in a (slightly depythonized variant) the Grazer Linux Tag 2013 ([slides and video](http://glt13-programm.linuxtage.at/events/198.de.html)).

holger krekel - packaging and testing with devpi and tox [EuroPython 2014] [24 July 2014] This talk discusses good ways to organise packaging and testing for Python projects. It walks through a per-company and an open source scenario and explains how to best use the "devpi-server" and "tox" for making sure you are delivering good and well tested and documented packages. As time permits, we also discuss in-development features such as real-time mirroring and search. ----- The talk discusses the following tools: - devpi-server for running an in-house or per-laptop python package server - inheritance between package indexes and from pypi.python.org public packages - the "devpi" client tool for uploading docs and running tests - running of tests through tox - summary view with two work flows: open source releases and in-house per-company developments - roadmap and in-development features of devpi and tox (The presenter is the main author of the tools in question).

Slavek Kabrda - Red Hat Loves Python [EuroPython 2014] [24 July 2014] Come learn about what Red Hat is doing with Python and the Python community, and how you can benefit from these efforts. Whether it is the new Python versions in Red Hat Enterprise Linux via the new Red Hat Software Collections, compatible Python cartridges in OpenShift Platform-as-a-Service (PaaS), or being the leading contributor to OpenStack, there's a lot going on at Red Hat. We're Pythonistas, too!

Floris Bruynooghe - Advanced Uses of py.test Fixtures [EuroPython 2014] [23 July 2014] One unique and powerful feature of py.test is the dependency injection of test fixtures using function arguments. This talk aims to walk through py.test's fixture mechanism gradually introducing more complex uses and features. This should lead to an understanding of the power of the fixture system and how to build complex but easily-managed test suites using them. ----- This talks will assume some basic familiarity with the py.test testing framework and explore only the fixture mechanism. It will build up more complex examples which will lead up to touching on other plugin features of py.test. It is expected people will be familiar with python features like functions as first-class objects, closures etc.

Maximilien Riehl - Practical PyBuilder [EuroPython 2014] [25 July 2014] PyBuilder is a software build tool written in pure python which mainly targets pure python applications. It provides glue between existing build frameworks, thus empowering you to focus on the big picture of the build process. It will be shown through demonstrations and samples how a simple, human-readable and declarative configuration can lead to an astonishingly well-integrated build process which will make maintainers, developers and newcomers happy. ----- # Why another build tool Starting up a simple python project with best practices still takes a lot of boilerplate and glueing (e.G. chaining unit tests and integration tests in the build process, adding a linter, measuring coverage, ...). It often results in extremely ugly homebrew scripts and edge-case solutions that are not reusable. There are even programs out there (e.G. cookiecutter) that encourage boilerplate code generation! # Build orchestration PyBuilder borrows from the *maven* idea of phases (packaging, verifying, publishing, ...) to set up a fully declarative and automated build that can be run locally and remotely (build servers) in the very same way. Rather than reinventing the wheel, it provides glue between existing solutions (like unittest, coverage, flake8, ...) through a simple but powerful plugin mechanism. # The talk After a more theoretical talk with a colleague at PyConDE 2013, I want to show how it's actually like to work with *PyBuilder*. This includes * starting up a project * running builds * using plugins * writing a plugin The demo code will be made available on GitHub and I'll probably have recordings prepped in case something goes wrong. Reviewer FAQ =============== ### How does PyBuilder compare to other existing solutions like zc.buildout? As opposed to solutions like zc.buildout which focus on the *building* of complex projects (many parts, complex dependencies) PyBuilder emphasizes the full build process for very simple projects. Undoubtedly, buildout is more powerful for building in that regard and there is no reason to switch to PyBuilder. However, for simple projects (a few packages, pure python) we believe that PyBuilder is better, especially if you're starting out with Python. The plugin architecture (as opposed to recipes) makes it easier to reason about what is going on. We are able to model dependencies between build phases (like "coverage" needing "unit tests" and "packaging" needing "integration tests") where recipes are not. It also seems (after looking through the recipes available for buildout) that we have more focus on QA as part of the build process (lint code, differentiate between unit/integration tests, code analysis, ...). There is also a special focus on having the build descriptor written in Python (with fluent interfaces where possible) so that it is possible to understand what the configuration is by reading plain english, as opposed to zc.buildout (where the configuration is an ini file, or SCons which is very make-oriented). In the end, a big difference between most build tools and PyBuilder is that PyBuilder is more about orchestration. We didn't reinvent packaging or linting, we simply use what is already there (setuptools, flake8, pymetrics, ...). This allows users to use the tools they want without having to do the integration themselves, and still get a nice, unified build process out of it. A simple example : In buildout, code analysis can be done with ``` [buildout] parts += code-analysis [code-analysis] recipe = plone.recipe.codeanalysis directory = ${buildout:directory}/src ``` This is not readable IMHO. In PyBuilder it can look like this : ``` use_plugin('python.flake8') project.set_property('flake8_include_test_sources', True) project.set_property('flake8_ignore', 'E501') project.set_property('flake8_break_build', True) ``` ### "It provides glue between existing build frameworks" - which ones ? Could you name (some at least) ? Currently there is only a plugin for building with distutils/setuptools. Should that change (e.G. new contender) it would be easy to switch using PyBuilder. Examples where glue is needed : * Glue setuptools + unittest, so that no distribution can be shipped if tests fail * Glue setuptools + coverage + unittest, so that no distribution can be shipped if the statement coverage is too low (configurable ofc) * Glue setuptools + pip so that cloned projects can be built with their dependencies without needing to pip install manually

Max Tepkeev - How we switched our 800+ projects from Apache to uWSGI [EuroPython 2014] [23 July 2014] During the last 7 years the company I am working for developed more than 800 projects in PHP and Python. All this time we were using Apache+nginx for hosting this projects. In this talk I will explain why we decided to switch all our projects from Apache+nginx to uWSGI+nginx and how we did that. ----- The talk will start from describing the setup we had for the last 7 years, i.e. Apache with mod_wsgi for Python projects and mod_php4/5 for PHP projects + nginx. I will explain why we used this setup for so long time, what problems we faced with this setup and what solutions we tried to solve them before switching to uWSGI. Then I will tell about uWSGI, what it is, how it works and what features it has. I will show the comparison of configuration files, how simple it is to configure uWSGI compared to Apache. Lastly I will explain how we managed to switch all our 800+ projects developed over the years in 2 different languages with 2 major versions changed (PHP4/5 and Python2/3), how this switch simplified our development and administration of this projects, the improvements we got in memory management and other areas. Of course I will concentrate mainly on our Python projects because it is EuroPython after all and not EuroPHP ;-)

Angel Ramboi - Gamers do REST [EuroPython 2014] [22 July 2014] An overview (sprinkled with implementation details and solutions to issues we encountered) of how Demonware uses Python and Django to build RESTful APIs and how we manage to reliably serve millions of gamers all over the world that play Activision-Blizzard’s successful franchises Call of Duty and Skylanders. Topics the presentation will touch: tech stack overview; API design; configuration handling; middleware usage for logging, metrics and error handling; authentication/authorization. ----- An overview (sprinkled with implementation details and solutions to issues we encountered) of how Demonware uses Python and Django to build RESTful APIs and how we manage to reliably serve millions of gamers all over the world that play Activision-Blizzard’s successful franchises Call of Duty and Skylanders. Topics the presentation will touch: - tech stack overview - API design - configuration handling - middleware usage for logging, metrics and error handling - authentication/authorization

Maciej Dziergwa - How to become an Agile company - case study [EuroPython 2014] [23 July 2014] The STX Next story has the classic arc of “zero to hero.” During the last 9 years, STX Next has grown from a small business with a handful of developers, to one of the biggest Python companies in Europe, and a leading proponent of agile and scrum methodologies. We feel that now is the best moment to share our experiences in implementing effective, agile development processes in a company of nearly 100 developers. Maciej Dziergwa will be discussing how he’s grown his business, what challenges there are today in python development, and how he plans to take his company to the next level. Especially we want to spread our ideas of building de-localized teams/frequently changing teams/teams with young members that learn rapidly making synergy effect. Join us during our Business Day on 23th July 2014! Remember that 2+2 can be much more than 4...

Travis Oliphant - The Continuum Platform: Advanced Analytics and Web-based Interactive Visualization for Enterprises [EuroPython 2014] [24 July 2014] The people at Continuum have been involved in the Python community for decades. As a company our mission is to empower domain experts inside enterprises with the best tools for producing software solutions that deal with large and quickly-changing data. The Continuum Platform brings the world of open source together into one complete, easy-to-manage analytics and visualization platform. In this talk, Dr. Oliphant will review the open source libraries that Continuum is building and contributing to the community as part of this effort, including Numba, Bokeh, Blaze, conda, llvmpy, PyParallel, and DyND, as well as describe the freely available components of the Continuum Platform that anyone can benefit from today: Anaconda, wakari.io, and binstar.org.

Frank - Managing the Cloud with a Few Lines of Python [EuroPython 2014] [23 July 2014] One of the advantages of cloud computing is that resources can be enabled or disabled dynamically. E. g. is an distributed application short on compute power one can easily add more. But who wants to do that by hand? Python is a perfect fit to control the cloud. The talk introduces the package Boto which offers an easy API to manage most of the Amazon Web Services (AWS) as well as a number of command line tools. First some usage examples are shown to introduce the concepts behind Boto. For that a few virtual hosts with different configurations are launched, and the use of the storage service S3 is briefly introduced. Based on that a scalable continuous integration system controlled by Boto is developed to show how easy all the required services can be used from Python. Most of the examples will be demonstrated during the talk. They should be easily adoptable for similar use cases or serve as an starting point for more different ones. ----- One of the advantages of cloud computing is that resources can be enabled or disabled dynamically. E. g. is an distributed application short on compute power one can easily add more. But who wants to do that by hand? Python is a perfect fit to control the cloud. The talk introduces the package Boto which offers an easy API to manage most of the Amazon Web Services (AWS) as well as a number of command line tools. First some usage examples are shown to introduce the concepts behind Boto. For that a few virtual instances with different configurations are launched, and the use of the storage service S3 is briefly introduced. Based on that a scalable continuous integration system controlled by Boto is developed to show how easy all the required services can be used from Python. Most of the examples will be demonstrated during the talk. They should be easily adoptable for similar use cases or serve as an starting point for more different ones.

Giles Thomas - An HTTP request's journey through a platform-as-a-service [EuroPython 2014] [23 July 2014] PythonAnywhere hosts tens of thousands of Python web applications, with traffic ranging from a couple of hits a week to dozens of hits a second. Hosting this many sites reliably at a reasonable cost requires a well-designed infrastructure, but it uses the same standard components as many other Python-based websites. We've built our stack on GNU/Linux, nginx, uWSGI, Redis, and Lua -- all managed with Python. In this talk we'll give a high-level overview of how it all works, by tracing how a request goes from the browser to the Python application and its response goes back again. As well as showing how a fairly large deployment works, we'll give tips on scaling and share a few insights that may help people running smaller sites discover how they can speed things up.

Maciej/Fabrizio Romano - Python Driven Company [EuroPython 2014] [25 July 2014] Adopting Python across a company brings extra agility and productivity not provided by traditional mainstream tools like Excel. This is the story of programmers teaching non-programmers, from different departments, to embrace Python in their daily work. ----- By introducing ipython notebook, pandas and the other data analysis packages that make python even more accessible and attractive, we attempted to adapt python as a core technology across our whole company. We’ve challenged the dominant position of Microsoft Excel and similar tools, and dared to replace it by pandas-powered ipython notebooks. During this transitional phase, we have been inspired and sometimes forced to develop multiple packages that extend pandas, numpy etc., in order to enable our colleagues, in other departments, to access all the data they need. Moreover, we are developing several high level functionalities for the notebook environment. The notebook environment is allowing us to be extremely responsive to the changes our users are asking for, since, for part of the work, we don’t have to go through the whole traditional development process. The talk focuses on challenges and problems we’ve solved and managed in order to achieve our long term goal of creating highly agile, data-driven non-tech teams, free from the constraints imposed by mainstream technologies, and all of this thanks to python.

Federico Marani - Scaling with Ansible [EuroPython 2014] [23 July 2014] Ansible is a powerful DevOps swiss-army knife tool, very easy to configure and with many extensions built-in. This talk will quickly introduce the basics of Ansible, then some real-life experience tips on how to use this tool, from setting up dev VMs to multi-server setups. ----- Infrastructure/Scaling is a topic really close to me, I'd like to have the chance to talk about how we set this up in the company I work for. Our infrastructure is around 10-15 servers, provisioned on different cloud providers, so a good size infrastructure. Presentation is going to be divided in 3 parts, first part is going to be focused on comparing sysadmin and devops, then there will be an introduction to the basic concepts of Ansible. I want to spend most on the last part, which is going to give some tips based on our experience with it. Many ideas will come from this presentation https://speakerdeck.com/fmarani/devops-with-ansible which i gave at DJUGL in London, with a longer session I will have more chances to delve into more detail, especially on how we use it, from vagrant boxes setup to AWS and DigitalOcean boxes, network configuration, software configurations, etc... I want to offer as many real-life tips as possible, without going too much offtopic as far as Ansible is concerned

Richard Wall - Twisted Names: DNS Building Blocks for Python Programmers [EuroPython 2014] [25 July 2014] In this talk I will report on my efforts to update the DNS components of Twisted and discuss some of the things I've learned along the way. I'll demonstrate the EDNS0, DNSSEC and DANE client support which I have been working on and show how these new Twisted Names components can be glued together to build novel DNS servers and clients. Twisted is an event-driven networking engine written in Python and licensed under the open source MIT license. It is a platform for developing internet applications. ----- # Description My talk will consist of four main sections. Given the 30 minute time constraint, I may shorten or drop the two introductory parts in favour of the narrative and demonstration of interesting new APIs and code examples in the final two parts. My experience of delivering a similar talk at PyconUK 2013 is that those are the parts that will most interest the audience and prompt most audience questions. Here are my proposed sections with rough time allocations and descriptions: ## Introducing Twisted Names (0-5) Twisted includes a comprehensive set of DNS components, collectively known as Twisted Names. - <https://twistedmatrix.com/trac/wiki/TwistedNames> I will begin the talk with a quick introduction to Twisted Names and its capabilities, including one or two simple code examples. ## Introducing My Project (0-5) With generous funding from The NLnet Foundation I am adding EDNS(0) and DNSSEC client support in Twisted Names, including full DNSSEC verification and DANE support. In the talk I will quickly summarise the steps taken and lessons learned in securing that funding, and hope to encourage the audience to seek funding to support there own pet OSS projects. ## What's New in Twisted Names / Project Progress Report (10) My project plan is divided into the following broad milestones. 1. EDNS(0) 1. OPT record 2. Extended Message object with additional EDNS(0) items 3. EDNS Client 2. RRSET handling 1. Canonical Form and Order of Resource Records 2. Receiving RRSETs 3. DNSSEC 1. New DNSSEC Records and Lookup Methods 2. Security-aware Non-validating Client 3. Validating Client 4. DANE 1. A twistd dns authoritative server capable of loading and serving TLSA records. 2. A Twisted web client Agent wrapper which performs TLSA lookup and verification of a server certificate. 3. A HostnameClientEndpoint which performs TLSA lookup and verification of a server certificate. 4. A command line tool for debugging TLSA records and for verifying a certificate file against a domain name. 5. A TLSA Record class for encoding and decoding TLSA bytes. 6. A TLSA lookup method which accepts port, protocol and hostname and constructs a suitable TLSA domain name. In the talk I will quickly outline these goals, report on my progress so far, and show running code examples to demonstrate the new APIs. ## Future Developments (5) The aim of my project is to lay foundations that will eventually allow end-to-end DNSSEC verification in all the core Twisted networking components, including Twisted Conch (SSH), Mail (SMTP, POP3), Perspective Broker (RPC), Web (HTTP, XML-RPC, SOAP), Words (XMPP, IRC). Additionally I hope that this foundation work will encourage the development of end-to-end DNSSEC verification in many of the Open Source and commercial projects built on top of Twisted. I will end the talk by outlining these exciting possibilities, and demonstrate some code examples that illustrate these possibilities. ## Q & A (5-10) I'm determined to leave at least five minutes at the end for audience questions. At PyconUK 2013 I was frustrated because I ran out of time and ended up answering questions outside the lecture theatre; questions which would have been interesting to the whole audience.

Daniel Pope - gevent: asynchronous I/O made easy [EuroPython 2014] [23 July 2014] gevent provides highly scalable asynchronous I/O without becoming a nest of callbacks, or even needing code changes. Daniel will explain how to get started with gevent, discuss patterns for its use and describe the differences with Twisted, Tornado and Tulip/asyncio. ----- It has been claimed "Callbacks are the new GOTO". Most asynchronous IO libraries use callbacks extensively. gevent (http://www.gevent.org) uses coroutines to provide highly scalable asynchronous I/O with a synchronous programming model that doesn't need code changes and callbacks. By elegantly monkey patching the Python standard library, both your code and all pure Python libraries become asynchronous too, making a separate collection of protocol implementations (in the style of Twisted) unnecessary. Code written like this is easier to understand, particularly for more junior developers. Crucially, IO errors can be raised at the right places. I will be introducing gevent's programming model, why it's easier, walk through simple code samples, and discuss experiences and metaphors for programming with it.

Ben Nuttall - Pioneering the Future of Computing Education [EuroPython 2014] [24 July 2014] How the Raspberry Pi Foundation are leading the way in the computing in schools revolution by providing affordable open and connectable hardware to people of all levels of experience. Now we have an education team, we're pushing forward with creating resources and training teachers to help deliver modern computing education around the world. All our learning resources are Creative Commons licensed and available on GitHub. We write materials that match the UK computing curriculum. ----- I'm Ben, from Raspberry Pi. I do development and outreach for the Foundation and I work with the rest of the education team to help make learning through computer science, coding and hardware hacking more accessible to all. In this talk I explain the Raspberry Pi story: its mission - the reason the Pi exists; what happened before release - getting the board in to production; what happened in the first two years - the community birth and growth; and what's coming next - education focus, new hardware and improved software. Python is the main language used (and advocated by us) in education with Raspberry Pi. We're creating learning resources to match up with the new UK computing curriculum, where we teach young people programming and computer science concepts with Python on Pi, and help teachers deliver quality material in the classroom to work towards the objectives the curriculum sets out to achieve. With Raspberry Pi we open up possibilities for connecting to the real world in an accessible way using the powerful, high level and human read/write -able language of Python. We work closely with the community: hobbyists organising Raspberry Jam events; educators teaching with Raspberry Pi; the software communities and their contributions - and we welcome any interested parties to get involved with helping us provide for the wider community.

Petr Viktorin - The Magic of Attribute Access [EuroPython 2014] [22 July 2014] Have you ever wondered how the "self" argument appears when you call a method? Did you know there is a general mechanism behind it? Come learn all about attributes and descriptors. ----- The first part of this talk will describe what exactly happens when you read or write an attribute in Python. While this behavior is, of course, explained in the Python docs, more precisely in the [Data model][1] section and [related][2] [writeups][3], the documentation gives one a "bag of tools" and leaves combining them to the reader. This talk, on the other hand, will present one chunk of functionality, the attribute lookup, and show how its mechanisms and customization options work together to provide the flexibility (and gotchas) Python provides. The topics covered will be: * method resolution order, with a nod to the C3 algorithm * instance-, class-, and metaclass-level variables * `__dict__` and `__slots__` * data/non-data descriptors * special methods (`__getattr__`, `__getattribute__`, `__setattr__`, `__dir__`) In the second part of the talk, I will show how to use the customization primitives explained before on several interesting and/or useful examples: * A proxy object using `__getattr__` * Generic desciptor - an ORM column sketch * the rudimentary `@property`, method, `staticmethod` reimplemented in pure Python (explained [here][2] and elsewhere), which lead to * SQLAlchemy's [`@hybrid_proprerty`][4] * Pyramid's deceptively simple memoizing decorator, [`@reify`][5] * An ["Unpacked" tuple properties][6] example to drive home the idea that descriptors can do more than provide attribute access (and mention weak dicts as a way to non-intrusively store data on an object) (These are subject to change as I compose the talk. Also some examples may end up interleaved with the theory.) Hopefully I'll have time to conclude with a remark about how Python manages to be a "simple language" despite having these relatively complex mechanisms. [1]: http://docs.python.org/3/reference/datamodel.html [2]: http://docs.python.org/3/howto/descriptor.html [3]: https://www.python.org/download/releases/2.3/mro/ [4]: http://docs.sqlalchemy.org/en/rel_0_9/orm/extensions/hybrid.html [5]: http://docs.pylonsproject.org/projects/pyramid/en/latest/api/decorator.html [6]: https://gist.github.com/encukou/9789993

Pieter Hintjens - Our decentralized future [EuroPython 2014 Keynote] [23 July 2014] Pieter will talk about the urgent push towards a decentralized future. As founder of the ZeroMQ community, he will explain the vision, design and reality of distributed software systems. He’ll explain his view on the community itself, also a highly decentralized “Living System”, as Hintjens calls it. Finally he’ll talk about edgenet, a model for a decentralized Internet.

Bob Ippolito - What can python learn from Haskell? [EuroPython 2014 Keynote] [21 July 2014] What can we learn from Erlang or Haskell for building reliable high concurrency services? Bob was involved in many Python projects but argues that for some domains there may be better methods found elsewhere. He started looking for alternatives back in 2006 when building high concurrency services at Mochi Media (originally with Twisted), which led him to the land of Erlang and later Haskell. Bob is going to talk about what he learned along the way. In particular, he’ll cover some techniques that are used in functional programming languages and how they can be used to solve problems in more performant, robust and/or concise ways than the standard practices in Python. He is also going to discuss some potential ways that the Python language and its library ecosystem could evolve accordingly. ----- What can we learn from Erlang or Haskell for building reliable high concurrency services? Bob was involved in many Python projects but argues that for some domains there may be better methods found elsewhere. He started looking for alternatives back in 2006 when building high concurrency services at Mochi Media (originally with Twisted), which led him to the land of Erlang and later Haskell. Bob is going to talk about what he learned along the way. In particular, he’ll cover some techniques that are used in functional programming languages and how they can be used to solve problems in more performant, robust and/or concise ways than the standard practices in Python. He is also going to discuss some potential ways that the Python language and its library ecosystem could evolve accordingly.

Emily Bache - Will I still be able to get a job in 2024 if I don't do TDD? [EuroPython 2014 Keynote] [22 July 2014] Geoffrey Moores's book "Crossing the chasm" outlines the difficulties faced by a new, disruptive technology, when adoption moves from innovators and visionaries into the mainstream. Test Driven Development is clearly a disruptive technology, that changes the way you approach software design and testing. It hasn't yet been embraced by everyone, but is it just a matter of time? Ten years from now, will a non-TDD practicing developer experience the horror of being labelled a technology adoption 'laggard', and be left working exclusively on dreadfully boring legacy systems? It could be a smart move to get down to your nearest Coding Dojo and practice TDD on some Code Katas. On the other hand, the thing with disruptive technologies is that even they can become disrupted when something better comes along. What about Property-Based Testing? Approval Testing? Outside-In Development? In this talk, I'd like to look at the chasm-crossing potential of TDD and some related technologies. My aim is that both you and I will still be able to get a good job in 2024. ----- TDD hasn't yet been embraced by everyone, but is it just a matter of time? Ten years from now, will a non-TDD practicing developer experience the horror of being labelled a technology adoption 'laggard', and be left working exclusively on dreadfully boring legacy systems? It could be a smart move to get down to your nearest Coding Dojo and practice TDD on some Code Katas. On the other hand, the thing with disruptive technologies is that even they can become disrupted when something better comes along. What about Property-Based Testing? Approval Testing? Outside-In Development? In this talk, I'd like to look at the chasm-crossing potential of TDD and some related technologies. My aim is that both you and I will still be able to get a good job in 2024.

Travis Oliphant - Python's Role in Big Data Analytics: Past, Present, and Future [EuroPython 2014 Keynote] [25 July 2014] Python has had a long history in Scientific Computing which means it has had the fundamental building blocks necessary for doing Data Analysis for many years. As a result, Python has long played a role in scientific problems with the largest data sets. Lately, it has also grown in traction as a tool for doing rapid Data Analysis. As a result, Python is the center of an emerging trend that is unifying traditional High Performance Computing with "Big Data" applications. In this talk I will discuss the features of Python and its popular libraries that have promoted its use in data analytics. I will also discuss the features that are still missing to enable Python to remain competitive and useful for data scientists and other domain experts. Finally, will describe open source projects that are currently occupying my attention which can assist in keeping Python relevant and even essential in Data Analytics for many years to come.

Constanze Kurz - One year of Snowden, what's next? [EuroPython 2014 Keynote] [21 July 2014] Since June 2013, disclosed by Edward Snowden, we learn more and more facts about American and British spies’ deep appetite for information, economic spying and the methods they use to collect data. They systematically tapped international communications on a scale that only few people could imagine. But what are the consequences for societies when they now know about the NSA metadata repository capable of taking in billions of "events" daily to collected and analyze? Is there a way to defend against an agency with a monstrous secret budget?

Christian Tismer/Anselm Kruis - Stackless: Recent advancements and future goals [EuroPython 2014] [23 July 2014] Stackless (formerly known as Stackless-Python) is an enhanced variant of the Python-language. Stackless is best known for its lightweight microthreads. But that's not all. In this talk Stackless core developers demonstrate recent advancements regarding multi-threading, custom-scheduling, debugging with Stackless and explain future plans for Stackless. ----- Stackless: Recent advancements and future goals ------------------------------------------------------- Since Python release 1.5 Stackless Python is an enhanced variant of C-Python. Stackless is best known for its addition of lightweight microthreads (tasklets) and channels. Less known are the recent enhancements that became available with Stackless 2.7.6. In this talk core Stackless developers demonstrate * The improved multi-threading support * How to build custom scheduling primitives based on atomic tasklet operations * The much improved debugger support * ... Stackless recently switched the new master repository from hg.python.org/stackless to bitbucket to allow for a more open development process. We'll summarise our experience and discuss our plans for the future development of Stackless. The talk will be help by Anselm Kruis and Christian Tismer. If we are lucky, we will also welcome Kristján Valur Jónsson from Iceland.

Sarah Mount - Message-passing concurrency for Python [EuroPython 2014] [22 July 2014] Concurrency and parallelism in Python are always hot topics. This talk will look the variety of forms of concurrency and parallelism. In particular this talk will give an overview of various forms of message-passing concurrency which have become popular in languages like Scala and Go. A Python library called python-csp which implements similar ideas in a Pythonic way will be introduced and we will look at how this style of programming can be used to avoid deadlocks, race hazards and "callback hell". ----- Concurrency and parallelism in Python are always hot topics. Early Python versions had a threading library to perform concurrency over operating system threads, Python version 2.6 introduced the multiprocessing library and Python 3.2 has introduced a futures library for asynchronous tasks. In addition to the modules in the standard library a number of packages such as gevent exist on PyPI to implement concurrency with "green threads". This talk will look the variety of forms of concurrency and parallelism. When are the different libraries useful and how does their performance compare? Why do programmers want to "remove the GIL" and why is it so hard to do? In particular this talk will give an overview of various forms of message-passing concurrency which have become popular in languages like Scala and Go. A Python library called python-csp which implements similar ideas in a Pythonic way will be introduced and we will look at how this style of programming can be used to avoid deadlocks, race hazards and "callback hell".

konarkmodi - Designing NRT(NearRealTime) stream processing systems: Using python with Storm and Kafka [EuroPython 2014] [22 July 2014] The essence of near-real-time stream processing is to compute huge volumes of data as it is received. This talk will focus on creating a pipeline for collecting huge volumes of data using Kafka and processing for near-real time computations using Storm.

konarkmodi - Design considerations while Evaluating, Developing, Deploying a distributed task processing system [EuroPython 2014] [23 July 2014] With the growing world of web, there are numerous use-cases which require tasks to be executed in an asynchronous manner and in a distributed fashion. Celery is one of the most robust, scalable, extendable and easy-to-implement frameworks available for distributed task processing. While developing applications using Celery, I have had considerable experience in terms of what design choices one should be aware of while evaluating an existing system or developing one's own system from scratch.

Chris Clauss - Pythonista: A full-featured Python environment for iOS devices [EuroPython 2014] [22 July 2014] The Pythonista app delivers a full-featured Python development experience on an iPad or an iPhone. This introduction to the app will provide a rapid overview of the Pythonista user experience, features and Community Forum. Then it will focus on a few source code examples of using the GPS to deliver real-time local weather, use the image library to manipulate images and convert documents, use the gyroscope to understand pitch, yaw, and roll, use Dropbox to backup and restore scripts, images, etc.

Dougal Matthews - Using asyncio (aka Tulip) for home automation [EuroPython 2014] [25 July 2014] This talk will cover the new asyncio library in Python 3.4 (also known as Tulip) and will use the area of home automation as a case study to explore its features. This talk will be based on code using Python 3.3+. Home automation is a growing area and the number of devices and potential applications is huge. From monitoring electricity usage to the temperature inside or outside your house to remote control of lights and other appliances the options are almost endless. However, managing and monitoring these devices is typically a problem that works best with event driven applications. This is where asnycio comes in, it was originally proposed in PEP 3156 by our BDFL, Guido van Rossum. Asyncio aims to bring a clear approach to the python ecosystem and borrows from a number of existing solutions to come up with something clean and modern for the Python stdlib. ----- This talk will cover the new asyncio library in Python 3.4 (also known as Tulip) and will use the area of home automation as a case study to explore its features. This talk will be based on code using Python 3.3+. Home automation is a growing area and the number of devices and potential applications is huge. From monitoring electricity usage to the temperature inside or outside your house to remote control of lights and other appliances the options are almost endless. However, managing and monitoring these devices is typically a problem that works best with event driven applications. This is where asnycio comes in, it was originally proposed in PEP 3156 by our BDFL, Guido van Rossum. Asyncio aims to bring a clear approach to the python ecosystem and borrows from a number of existing solutions to come up with something clean and modern for the Python stdlib. This talk will introduce asyncio and use it within the context of home automation and dealing with multiple event driven devices. Therefore we will cover asyncio and the lessions learned from using different devices in this context. Some of the devices that will be used include: - Raspberry Pi - RFXCom's RFXtrx, USB serial tranciever. - Owl CM160 electricity tracker. - Oregon scientific thermometers. - Foscam IP cameras. This talk will also briefly cover the previous solution I used which was developed with Twisted and compare it briefly with my new code using asyncio.

Marc-Andre Lemburg - Advanced Database Programming with Python [EuroPython 2014] [25 July 2014] The Python DB-API 2.0 provides a direct interface to many popular database backends. It makes interaction with relational database very straight forward and allows tapping into the full set of features these databases provide. The talk will cover advanced database topics which are relevant in production environments such as locks, distributed transactions and transaction isolation. ----- The Python DB-API 2.0 provides a direct interface to many popular database backends. It makes interaction with relational database very straight forward and allows tapping into the full set of features these databases provide. The talk will cover advanced database topics which are relevant in production environments such as locks, distributed transactions and transaction isolation. ---- The talk will give an in-depth discussion of advanced database programming topics based on the Python DB-API 2.0: locks and dead-locks, two-phase commits, transaction isolation, result set scrolling, schema introspection and handling multiple result sets. Talks slides are available on request.

Heiko - The Shogun Machine Learning Toolbox [EuroPython 2014] [24 July 2014] We present the Shogun Machine Learning Toolbox, a framework for Machine Learning, which is the art of finding structure in data, with applications in object recognition, brain-computer interfaces, robotics, stock-prices prediction, etc. We give a gentle introduction to ML and Shogun's Python interface, focussing on intuition and visualisation. ----- We present the Shogun Machine Learning Toolbox, a unified framework for Machine Learning algorithms. Machine Learning (ML) is the art of finding structure in data in an automated way and has given rise to a wide range of applications such as recommendation systems, object recognition, brain-computer interfaces, robotics, predicting stock prices, etc. Our toolbox offers extensive bindings with other software and computing languages, Python being the major target. The library was initiated in 1999 and remained under heavy development henceforth. In addition to its mature core-framework, Shogun offers state-of-the-art techniques based on latest ML research. This is partly made possible by the 21 Google Summer of Code projects (5+8+8 since 2011) that our students successfully completed. Shogun's codebase has >20k commits made by >100 contributors representing >500k lines of code. While its core is written in C++, a unique of technique for generating interfaces allows usage from a wide range of target languages -- under the same syntax. This includes in particular Python, but also Matlab/Octave, Java, C#, R, ruby, and more. We believe that users should be able to choose their favourite language rather than us dictating this choice. The same applies for supported OS (Linux, Mac, Win). Shogun is part of Debian Linux. Features of Shogun include most classical ML methods such as classification, regression, dimensionality reduction, clustering, etc, most of them in different flavours. All implemented algorithms in Shogun work on a modular data representation, which allows to easily switch between different sorts of objects as for example strings or matrices. Common ML-tasks and data IO can be carried under a unified interface. This is also true for the various external open-source libraries that are embedded within Shogun. Code examples are provided for all implemented algorithms. The main and most complete set of examples is in the Python language. In addition, in order to push usage of Shogun in education at universities, we recently started adding more illustrative IPython notebooks. A growing list of statically rendered versions are readily available from our [website](http://www.shogun-toolbox.org/page/documentation/notebook) and implement a cross-over of tutorial-style explanations, code, and visualization examples. We even took this up a notch and started building our own IPython-notebook server with Shogun installed in the cloud at (try cloud button in notebook view) . This allows users to try Shogun without installation via the IPython notebook web interface. All example notebooks can be loaded, interactively modified, and executed. In addition, using the Python Django framework, we built a collection of interactive web-demos where users can play around with basic ML algorithms, [demos](http://www.shogun-toolbox.org/page/documentation/demo) In the proposed talk, we will give a gentle and general introduction to ML and the core functionality of Shogun, with a focus on its Python interface. This includes solving basic ML tasks such as classification and regression and some of the more recent features, such as last year's GSoC projects and their IPython notebook writeups. ML material will be presented with a focus on intuition and visualisation and no previous familiarity with ML methods is required. ## Key points in the talk * What are the goals in ML? * Example problems in ML (classification, regression, clustering) * Some basic algorithm ideas * Focus on Visualisation, not Maths ## Intended Audience * All people dealing with data (data scientists, big-data hackers) who are looking for tools to deal with it * People with a general interest but no education in Machine Learning * People interested in the technology behind Shogun (swig, cloud notebook server, web-demos) * People from the ML community (scipy-stack) * ML scientists/Statisticians ## Code examples * [Classification](https://github.com/shogun-toolbox/shogun/blob/develop/examples/undocumented/python_modular/classifier_libsvm_modular.py) * [Clustering](https://github.com/shogun-toolbox/shogun/blob/develop/examples/undocumented/python_modular/graphical/em_2d_gmm.py) * [Source seperation](https://github.com/shogun-toolbox/shogun/blob/develop/examples/undocumented/python_modular/graphical/converter_jade_bss.py) * [IPython notebook examples](http://www.shogun-toolbox.org/page/documentation/notebook) ### Slide examples See our Europython 2010 [slides](https://www.drop

Andreas Pelme - Introduction to pytest [EuroPython 2014] [23 July 2014] pytest is a full featured testing tool that makes it possible to write “pythonic” tests. This talk will introduce pytest and some of its unique and innovative features. It will help you get started with pytest for new or existing projects, by showing basic usage and configuration. ----- This talk will show introduce pytest and show some unique and innovative features. It will show how to get started using it and some of the most important features. One of these features is the ability to write tests in a more “pythonic” way by using the assert statement for assertions. Another feature in pytest is fixtures – a way to handle test dependencies in a structured way. This talk will introduce the concept of fixtures and show how they can be used. No previous knowledge of pytest is required – this talk is for people who are new to testing or has experience with other Python testing tools such as unittest or Nose.

Markus Zapke-Gründemann - Writing multi-language documentation using Sphinx [EuroPython 2014] [23 July 2014] How to write multi-language documentation? What tools can you use? What mistakes should you avoid? This talk is based on the experiences I gathered while working on several multi-language documentation projects using Sphinx. ----- Internationalized documentation is a fairly new topic. And there are different approaches to do this. I will talk about how Sphinx internationalization support works, which tools and services I use and how to organize the translation workflow in an Open Source project. Finally I will have a look at what the future of internationalization in Sphinx might bring.

Reimar Bauer - pymove3D - Python moves the world - Attractive programming for young people. [EuroPython 2014] [22 July 2014] The second time a contest for schoolar students is organized by the Python Software Verband e.V.. It is about to write a Python program that is executable in Blender using its 3D capabilities. The talk overall gives an overview what experience we got by these ideas and how we want to continue.

Robert Lehmann - Teaching Python [EuroPython 2014] [22 July 2014] Using Python in bringing people closer to programming has been popular for a while. But what are the most effective ways to do so? The OpenTechSchool reports. ----- Python has been used in educational programmes ever since. With a bandwidth that large, navigating the landscape of Python tutorials is hard indeed. This talk will look at successful Python teaching material. From the numerous iterations our material has gone through, we draw conclusions on what's crucial in teaching Python. It will introduce how the OpenTechSchool is teaching Python and what measures it found most effective in spreading programming in general and Python in particular. Among these are rapid feedback, supervised learning, localization, and knowing your target audience. The author is a member of the OpenTechSchool, a free community initiative which offers Python workshop on a number of topics. Some of the workshops have been running for more than two years now. He has written the first versions of "Python for beginners," a workshop which has been used in many cities to teach Python to programming novices.

Mislav Stipetic/Darko Ronić - Mobile Games to the Cloud With Python [EuroPython 2014] [22 July 2014] When a mobile game development company decides to switch to a more cloud based development it is faced with obstacles different from those it’s used to on mobile devices. This talk explains how Python provided us with most of the infrastructure for this task and how a Python game backend was built as a result. ----- #### The Talk This talk has two goals. Showing the audience the lessons we learned during a project which moved a simple mobile game to a server backend is our first intention. In addition to that we want to describe how such a system works in a real life example, to show which problems and which requirements arise in its creation. When the audience leaves the talk they will know how a real-life mobile game uses Python for powering the backend servers. #### The Problem Most of the game development for mobile devices is focused on running the game on the device. The game designers and game developers play a primary role in creating the product. The server backend plays a supporting role providing a multiplayer or social experience to the users. Indeed, at Nanobit Ltd., things were also done that way. We had a small Python infrastructure built around Django which provided a small portion of multiplayer experience for the players. The majority of development was still focused on playing the game on the device. That way of thinking was put to test when we decided to center our future games around the multiplayer experience. Due to the fact that our infrastructure at the time was not enough for what we had in mind, we had to start from scratch. The decision was made to use Python as the center of our new infrastructure. In order to achieve it, a server backend was needed that would allow the game to be played “in the cloud” with the device only being a terminal to the player. Most of the game logic would have to be processed in the cloud which meant that each player required a constant connection to the backend and with over 100.000 players in our previous games that presented a challenge. How to build an infrastructure which can support that? Since every user action had to be sent to the backend how to process thousands of them quick enough? Those problems were big and were just the start. #### The Solution The design of the backend lasted for a couple of months and produced a scalable infrastructure based on “workers” developed in Python, “web servers” that use Tornado and a custom message queue which connected the two. The storage part is a combination of Riak and Redis. Since the backend is scalable new workers and new web servers had to be deployed easily so an orchestration module was build using Fabric. The scalability and launching of new workers and web servers was achieved using Docker for creation and deployment of containers. Each container presents one module of the system (worker, web server, queue). The end result can now support all of our future games and only requires the game logic of each game to be added to the workers. #### The Technologies * Python for coding the game logic, web servers. More than 90% of the system was written in Python. * Fabric * SQLAlchemy * Riak * Redis * ZeroMQ * nginx * Docker * Websockets * AWS #### The Lessons Learned * How to tune the backend to handle the increasing number of active players. * How to tackle the problem of frequent connection dropping and reachability issues of poor mobile device Internet connection in Tornado with a little help of Redis. * How to prevent users from trying to outsmart the system by denying illegal moves. * How to enable game profile syncing and live updating. * Improving the performance of workers by prioritizing data being stored to databases (Riak, SQL). * New issues and lessons show up all the time so there will definitely be more of them by the time of the conference. #### Basic Outline 1. Intro (5 min) 1. Who are we? 2. How was Python used in our previous games 3. Why we decided to change it all 2. Requirements (6 min) 1. What was the goal of creating the game backend 2. Why was Python our first choice 3. Python backend (14 min) 1. The architecture of the backend 2. Which technologies did we use and how were they connected together 3. How the backend handles the game logic 4. Lessons learned 4. Questions & Answers (5 min)

Josef Heinen - Scientific Visualization with GR [EuroPython 2014] [25 July 2014] Python developers often get frustrated when managing visualization packages that cover the specific needs in scientific or engineering environments. The GR framework could help. GR is a library for visualization applications ranging from publication-quality 2D graphs to the creation of complex 3D scenes and can easily be integrated into existing Python environments or distributions like Anaconda. ----- Python has long been established in software development departments of research and industry, not least because of the proliferation of libraries such as *SciPy* and *Matplotlib*. However, when processing large amounts of data, in particular in combination with GUI toolkits (*Qt*) or three-dimensional visualizations (*OpenGL*), it seems that Python as an interpretative programming language may be reaching its limits. --- *Outline* - Introduction (1 min) - motivation - GR framework (2 mins) - layer structure - output devices and capabilities - GR3 framework (1 min) - layer structure - output capabilities (3 mins) - high-resolution images - POV-Ray scenes - OpenGL drawables - HTML5 / WebGL - Simple 2D / 3D examples (2 min) - Interoperability (PyQt/PySide, 3 min) - How to speed up Python scripts (4 mins) - Numpy - Numba (Pro) - Animated visualization examples (live demos, 6 mins) - physics simulations - surfaces / meshes - molecule viewer - MRI voxel data - Outlook (1 min) *Notes* Links to similar talks, tutorials or presentations can be found [here][1]. Unfortunately, most of them are in German language. The GR framework has already been presented in a talk at PyCon DE [2012][2] and [2013][3], during a [poster session][4] at PyCon US 2013, and at [PythonCamps 2013][5] in Cologne. The slides for the PyCon.DE 2013 talk can be found [here][6]. As part of a collaboration the GR framework has been integrated into [NICOS][7] (a network-based control system completely written in Python) as a replacement for PyQwt. [1]: http://gr-framework.org/ [2]: https://2012.de.pycon.org/programm/schedule/sessions/54 [3]: https://2013.de.pycon.org/schedule/sessions/45/ [4]: https://us.pycon.org/2013/schedule/presentation/158/ [5]: http://josefheinen.de/rasberry-pi.html [6]: http://iffwww.iff.kfa-juelich.de/pub/doc/PyCon_DE_2013 [7]: http://cdn.frm2.tum.de/fileadmin/stuff/services/ITServices/nicos-2.0/dirhtml/

andrea crotti - Metaprogramming, from decorators to macros [EuroPython 2014] [24 July 2014] Starting off with the meaning of metaprogramming we quickly dive into the different ways Python allows this. First we talk about class and functions decorators, when decorators are not enough anymore we'll explore the wonders of metaclasses. In the last part of the talk we'll talk about macros, first in Lisp and then using the amazing macropy library. ----- This talk is a journey in the wonderful world of metaprogramming. We start off with the meaning of metaprogramming and what it can be used for. Then we look at what can be done in Python, introducing function and class decorators. When decorators are not enough anymore we move to the black magic of metaclasses, showing how we can implemement a simple Django-like model with them. In the bonus track we'll talk about macros, as the ultimate metaprogramming weapon, showing briefly how Lisp macros work and introducing the amazing [macropy library](https://github.com/lihaoyi/macropy).

Dave Halter - Identifying Bugs Before Runtime With Jedi [EuroPython 2014] [23 July 2014] Finding bugs before runtime has been an incredibly tedious task in Python. Jedi is an autocompletion library with interesting capabilities: It understands a lot of the dynamic features of Python. I will show you how we can use the force of (the) Jedi to identify bugs in your Python code. It's not just another pylint. It's way better. ----- Jedi is an autocompletion library for Python that has gained quite a following over the last year. There are a couple of plugins for the most popular editors (VIM, Sublime, Emacs, etc.) and mainstream IDEs like Spyder are switching to Jedi. Jedi basically tries to redefine the boundaries of autocompletion in dynamic languages. Most people still think that there's no hope for decent autocompletion in Python. This talk will try to argue the opposite, that decent autocompletion is very close. While the first part will be about Jedi, the second part of this talk will discuss the future of dynamic analysis. Dynamic Analysis is what I call the parts that static analysis doesn't cover. The hope is to generate a kind of "compiler" that doesn't execute code but reports additional bugs in your code (AttributeErrors and the like). I still have to work out the details of the presentation. I also have to add that Jedi I'm currently working full-time on Jedi and that there's going to be some major improvements until the conference. Autocompletion and static/dynamic analysis as well as refactoring are hugely important tools for a dynamic language IMHO, because they can improve the only big disadvantage compared to static languages: Finding bugs before running your tool.

Valerio Maggio - Scikit-learn to "learn them all" [EuroPython 2014] [24 July 2014] Scikit-learn is a powerful library, providing implementations for many of the most popular machine learning algorithms. This talk will provide an overview of the "batteries" included in Scikit-learn, along with working code examples and internal insights, in order to get the best for our machine learning code. ----- **Machine Learning** is about *using the right features, to build the right models, to achieve the right tasks* [[Flach, 2012]][0] However, to come up with a definition of what actually means **right** for the problem at the hand, it is required to analyse huge amounts of data, and to evaluate the performance of different algorithms on these data. However, deriving a working machine learning solution for a given problem is far from being a *waterfall* process. It is an iterative process where continuous refinements are required for the data to be used (i.e., the *right features*), and the algorithms to apply (i.e., the *right models*). In this scenario, Python has been found very useful for practitioners and researchers: its high-level nature, in combination with available tools and libraries, allows to rapidly implement working machine learning code without *reinventing the wheel*. [**Scikit-learn**](http://scikit-learn.org/stable/) is an actively developing Python library, built on top of the solid `numpy` and `scipy` packages. Scikit-learn (`sklearn`) is an *all-in-one* software solution, providing implementations for several machine learning methods, along with datasets and (performance) evaluation algorithms. These "batteries" included in the library, in combination with a nice and intuitive software API, have made scikit-learn to become one of the most popular Python package to write machine learning code. In this talk, a general overview of scikit-learn will be presented, along with brief explanations of the techniques provided out-of-the-box by the library. These explanations will be supported by working code examples, and insights on algorithms' implementations aimed at providing hints on how to extend the library code. Moreover, advantages and limitations of the `sklearn` package will be discussed according to other existing machine learning Python libraries (e.g., [`shogun`](http://shogun-toolbox.org "Shogun Toolbox"), [`pyML`](http://pyml.sourceforge.net "PyML"), [`mlpy`](http://mlpy.sourceforge.net "MLPy")). In conclusion, (examples of) applications of scikit-learn to big data and computational intensive tasks will be also presented. The general outline of the talk is reported as follows (the order of the topics may vary): * Intro to Machine Learning * Machine Learning in Python * Intro to Scikit-Learn * Overview of Scikit-Learn * Comparison with other existing ML Python libraries * Supervised Learning with `sklearn` * Text Classification with SVM and Kernel Methods * Unsupervised Learning with `sklearn` * Partitional and Model-based Clustering (i.e., k-means and Mixture Models) * Scaling up Machine Learning * Parallel and Large Scale ML with `sklearn` The talk is intended for an intermediate level audience (i.e., Advanced). It requires basic math skills and a good knowledge of the Python language. Good knowledge of the `numpy` and `scipy` packages is also a plus. [0]: http://goo.gl/BnhoHa "Machine Learning: The Art and Science of Algorithms that Make Sense of Data, *Peter Flach, 2012*"

synasius - How to make a full fledged REST API with Django OAuth Toolkit [EuroPython 2014] [22 July 2014] World is going mobile and the need of a backend talking with your apps is getting more and more important. What if I told you writing REST APIs in Python is so easy you don’t need to be a backend expert? Take generous tablespoons of Django, mix thoroughly with Django REST Framework and dust with Django OAuth Toolkit to bake the perfect API in minutes.

Thomas Wiecki - Probabilistic Programming in Python [EuroPython 2014] [24 July 2014] Probabilistic Programming allows flexible specification of statistical models to gain insight from data. The high interpretability and ease by which different sources can be combined has huge value for Data Science. PyMC3 features next generation sampling algorithms, an intuitive model specification syntax, and just-in-time compilation for speed, to allow estimation of large-scale probabilistic models. ----- Probabilistic Programming allows flexible specification of statistical models to gain insight from data. Estimation of best fitting parameter values, as well as uncertainty in these estimations, can be automated by sampling algorithms like Markov chain Monte Carlo (MCMC). The high interpretability and flexibility of this approach has lead to a huge paradigm shift in scientific fields ranging from Cognitive Science to Data Science and Quantitative Finance. PyMC3 is a new Python module that features next generation sampling algorithms and an intuitive model specification syntax. The whole code base is written in pure Python and Just-in-time compiled via Theano for speed. In this talk I will provide an intuitive introduction to Bayesian statistics and how probabilistic models can be specified and estimated using PyMC3.

Simon Cross - Conversing with people living in poverty [EuroPython 2014] [24 July 2014] Vumi is a text messaging system designed to reach out to those in poverty on a massive scale via their mobile phones. It's written in Python using Twisted. This talk is about how and why we built it and how you can join us in making the world a better place. ----- 43% of the world's population live on less than €1.5 per day. The United Nations defines poverty as a "lack of basic capacity to participate effectively in society". While we often think of the poor as lacking primarily food and shelter, the UN definition highlights their isolation. They have the least access to society's knowledge and services and the most difficulty making themselves and their needs heard in our democracies. While smart phones and an exploding ability to collect and process information are transforming our access to knowledge and the way we organize and participate in our societies, those living in poverty have largely been left out. This has to change. Basic mobile phones present an opportunity to effect this change [3]. Only three countries in the world have fewer than 65 mobile phones per 100 people [4]. The majority of these phones are not Android or iPhones, but they do nevertheless provide a means of communication -- via voice calls, SMSes [6], USSD [7] and instant messaging. By comparison, 25 countries have less than 5% internet penetration [5]. Vumi [1] is an open source text messaging system designed to reach out to those in poverty on a massive scale via their mobile phones. It's written in Python using Twisted. Vumi is already used to: * provide Wikipedia access over USSD and SMS in Kenya [8]. * register a million voters in Libya [10]. * deliver health information to mothers in South Africa [9]. * prevent election violence in Kenya [11]. This talk will cover: * a brief overview of mobile networking and cellphone use in Africa * why we built Vumi * the challenges of operating in unreliable environments * an overview of Vumi's features and architecture * how you can help! Vumi features some cutting edge design choices: * horizontally scalable Twisted processes communicating using RabbitMQ. * declarative data models backed by Riak. * sharing common data models between Django and Twisted. * sandboxing hosted Javascript code from Python. Overview of challenges Vumi addresses: *Scalability*: Vumi needs to support both small scale applications (demos, pilot projects, applications tailored for a particular community) and large ones (things that everyone within a country might use). We address this using Twisted workers that exchange messages via RabbitMQ and store data in Riak. Having projects share RabbitMQ and Riak instances significantly reduces the overhead for small projects (e.g. its not cost effective to launch the recommended minimum of 5 Riak servers for a small project). *Barriers to entry*: Often the people with good ideas don't have access to one of many things needed to run a production system themselves, e.g. capital, time, stable infrastructure. We address this by providing a hosted Vumi instance that runs sandboxed Javascript applications. All the application author needs is their idea, the ability to write Javascript and upload it to our servers. The target audience here is African entrepreneurs at incubator spaces like iHub (Nairobi), kLab (Kigali), BongoHive (Lusaka) and JoziHub (Johannesburg). *Unreliable third-party systems*: It's one thing for parts of ones own system to go down, it's another for crucial third-party systems to go down. Vumi takes an SMTP-like approach to solving this and uses persistent queues so that messages can back up in the queue while third-party systems are down and be processed when they become available again. We also feedback information on whether third-party messaging systems have accepted or reject messages to the application that initiated them. Vumi is developed by the Praekelt Foundation [2] (and individual contributors!). [1]: <http://vumi.org/> "Vumi" [2]: <http://praekeltfoundation.org/> "Praekelt Foundation" [3]: <http://www.youtube.com/watch?v=0bXjgx4J0C4#t=20> "Spotlight on Africa" [4]: <http://en.wikipedia.org/wiki/List_of_countries_by_number_of_mobile_phones_in_use> [5]: <http://en.wikipedia.org/wiki/List_of_countries_by_number_of_Internet_users> [6]: <http://en.wikipedia.org/wiki/Short_Message_Service> [7]: <http://en.wikipedia.org/wiki/Unstructured_Supplementary_Service_Data [8]: <http://blog.praekeltfoundation.org/post/65981723628/wikipedia-zero-over-text-with-praekelt-foundation> [9]: <http://blog.praekeltfoundation.org/post/65042080515/mama-launches-healthy-family-nutrition-programme> [10]: <http://www.libyaherald.com/2014/01/01/over-one-million-register-for-constitutional-elections-on-final-sms-registration-day

Magdalena Rother - How to become a software developer in science? [EuroPython 2014] [24 July 2014] My path from 'Hello world' to software development was long and hard. The approach I learned during my research may help you to create high quality software and improve as a developer. The talk covers how you can benefit from your non-IT knowledge, atomize your project and how collaboration accelerates your learning. ----- **Goal**: give practical tools for improving skills and software quality to people with a background other than IT. Eight years ago, as a plant biologist, I knew almost nothing about programming. When I took a course in python programming, I found myself so fascinated that it altered my entire career. I became a scientific software developer. It was long and hard work to get from the level of 'Hello world' to the world of software development. The talk will cover how to embrace a non-IT education as a strength, how and why to atomize programming tasks and the importance of doing side projects. ### 1. Embrace your background Having domain specific knowledge from a field other than IT helps you to communicate with the team, the users and the group leader. It prevents misunderstandings and helps to define features better. A key step you can take is systematically apply the precise domain specific language to the code e.g when naming objects, methods or functions. Another is to describe the underlying scientific process step by step as a Use Case and write it down in pseudocode. ### 2. Atomisation Having a set of building block in your software helps to define responsibilities clearly. Smaller parts are easier to test, release and change. Modular design makes the software more flexible and avoids the Blob and Lava Flow Anti-Patterns. When using object oriented programming a rule of thumb is that an object (in Python also a method) does only one thing. You can express this Single Responsibility Principle as a short sentence for each module. Another practical action is to introduce Design Patterns that help to decouple data and its internal representation. As a result, your software becomes more flexible. ### 3. Participating in side projects Learning from others is a great opportunity to grow. Through side projects you gain a fresh perspective and learn about best practices in project management. You gain new ideas for improvement and become aware of difficulties in your own project. You can easily participate in a scientific project by adding a small feature, writing a test suite or provide a code review on a part of a program. Summarizing, in scientific software development using domain-specific knowledge, atomisation of software, and participation in side projects are three things that help to create high quality software and to continuously improve as a developer. The talk will address challenges in areas where science differs from the business world. It will present general solution one might use for software developed in a scientific environment for research projects rather then discussing particular scientific packages. ### Qualifications During my PhD I developed a software on 3D RNA modeling (www.genesilico.pl/moderna/) that resulted in 7 published articles. I am coauthor on a paper on bioinformatic software development. Currently I am actively developing a system biology software in Python at the Humboldt University Berlin (www.rxncon.org).

Ralph Heinkel - Combining the powerful worlds of Python and R [EuroPython 2014] [25 July 2014] Although maybe not very well known in the Python community there exists a powerful statistical open-source ecosystem called R. Mostly used in scientific contexts it provides lots of functionality for doing statistical analysis, generation of various kinds of plots and graphs, and much, much more. The triplet R, Rserve, and pyRserve allows the building up of a network bridge from Python to R: Now R-functions can be called from Python as if they were implemented in Python, and even complete R scripts can be executed through this connection. ----- pyRserve is a small open source project originally developed to fulfill the needs of a German biotech company to do statistical analysis in a large Python-based Lab Information Management System (LIMS). In contrast to other R-related libraries like RPy where Python and R run on the same host, pyRserve allows the distribution of complex operations and calculations over multiple R servers across the network. The aim of this talk is to show how easily Python can be connected to R, and to present a number of selected (simple) code examples which demonstrate the power of this setup.

Ashikaga - Python for Zombies: 15.000 enrolled in the first Brazilian MOOC to teach Python [EuroPython 2014] [24 July 2014] Experiences of how we spread the Python community in Brazil with a non english MOOC (Massive Open Online Course) to teach programming. Hacking basic modules and classes to obtain the "Answer to the Ultimate Question of Life, the Universe, and Everything". A funny way to teach programming.

Carl Crowder - Automatic code reviews [EuroPython 2014] [23 July 2014] A lot of great Python tools exist to analyse and report on your codebase, but they can require a lot of initial set up to be useful. Done right, they can be like an automatic code review. This talk will explain how to set up and get the best out of these tools, especially for an existing, mature codebase. ----- Static analysis tools are a great idea in theory, but are not often really used in practice. These tools usually require quite a lot of initial effort to get set up in a way which produces meaningful output for you or your organisation's particular coding style and values. As a result, it's common to see initial enthusiasm replaced by ignoring the tools. Such tools can be incredibly beneficial however, and even go so far as to provide an automatic code review, and this talk will explain what kind of benefits you can get from the tools, as well as explain what you can and cannot expect. This talk is aimed at experienced developers who are interested in improving their coding practices but who have either never tried static analysis tools, or who have not seen the upsides. It will hopefully also be useful to people who do use the tools, perhaps introducing them to new tools or concepts they were not aware of yet.

Stefan Schwarzer - Support Python 2 and 3 with the same code [EuroPython 2014] [24 July 2014] Your library supports only Python 2, - but your users keep nagging you about Python 3 support? As Python 3 gets adopted more and more, users ask for Python 3 support in existing libraries for Python 2. Although there are several approaches, this talk focuses on using the very same code for a Python 2 and a Python 3 version. The talk discusses the main problems when supporting Python 3 and best practices to apply for compatibility with Python 2 and 3. ----- Your library supports only Python 2, - but your users keep nagging you about Python 3 support? As Python 3 gets adopted more and more, users ask for Python 3 support in existing libraries for Python 2. This talk mentions some approaches for giving users a Python 3 version, but will quickly focus on using the very same code for a Python 2 and a Python 3 version. This is much easier if you require Python 2.6 and up, and yet a bit easier if you require Python 3.3 as the minimum Python 3 version. The talk discusses main problems when supporting Python 3 (some are easily solved): * `print` is a function. * More Python APIs return iterators that used to return lists. * There's now a clear distinction between bytes and unicode (text) strings. * Files are opened as text by default, requiring an encoding to apply on reading and writing. The talk also explains some best practices: * Start with a good automatic test coverage. * Deal with many automatic conversions with a one-time 2to3 run. * Think about how your library should handle bytes and unicode strings. (Rule of thumb: Decode bytes as early as possible; encode unicode text as late as possible.) * Should you break compatibility with your existing Python 2 API? (Yes, if there's no other way to design a sane API for Python 2 and 3. If you do it, raise the first part of the version number.) * Try to keep code that's different for Python 2 and 3 minimal. Put code that needs to be different for Python 2 and 3 into a `compat` module. Or use third-party libraries like `six` or `future`. Finally, the talk will mention some helpful resources on the web.

Jyrki Pulliainen - Packaging in packaging: dh-virtualenv [EuroPython 2014] [24 July 2014] Deploying your software can become a tricky task, regardless of the language. In the spirit of the Python conferences, every conference needs at least one packaging talk. This talk is about dh-virtualenv. It's a Python packaging tool aimed for Debian-based systems and for deployment flows that already take advantage of Debian packaging with Python virtualenvs ----- [Dh-virtualenv][1] is an open source tool developed at Spotify. We use it to ease deploying our Python software to production. We built dh-virtualenv as a tool that fits our existing continuous integration flow with a dedicated sbuild server. As we were already packaging software in Debian packages, the aim of dh-virtualenv was to make transition to virtualenv based installations as smooth as possible. This talk covers how you can use dh-virtualenv to help you deploy your software to production, where you are already running a Debian-based system, such as Ubuntu, and what are the advantages and disadvantages of the approach over other existing and popular techniques. We will discuss the deploying as a problem in general, look into building a dh-vritualenv-backed package, and in the end, look into how dh-virtualenv was actually made. Goal is that after this presentation you know how to make your Debian/Ubuntu deployments easier! [dh-virtualenv][1] if fully open sourced, production tested software, licensed under GPLv2+ and available in Debian testing and unstable. More information of it is also available in our [blogpost][2]. Talk outline: 1. Introduction & overview (3min) * Who am I? * Why am I fiddling with Python packaging? * What do you get out of this talk? 2. Different shortcomings of Python deployments (5min) * Native system packages * Virtualenv based installations * Containers, virtual machine images 3. dh-virtualenv (10 min) * What is dh-virtualenv? * Thought behind dh-virtualenv * Advantages over others * Requirements for your deployment flow * Short intro to packaging Sentry with dh-virtualenv 4. How is it built? (10 min) * Debian package building flow primer * How dh-virtualenv fits that flow * What does it do build time and why? [1]:http://github.com/spotify/dh-virtualenv [2]:http://labs.spotify.com/2013/10/10/packaging-in-your-packaging-dh-virtualenv/

Francisco Fernández Castaño - Graph Databases, a little connected tour [EuroPython 2014] [23 July 2014] There are many kinds of NoSQL databases like, document databases, key-value, column databases and graph databases. In some scenarios is more convenient to store our data as a graph, because we want to extract and study information relative to these connections. In this scenario, graph databases are the ideal, they are designed and implemented to deal with connected information in a efficient way. ----- There are many kinds of NoSQL databases like, document databases, key-value, column databases and graph databases. In some scenarios is more convenient to store our data as a graph, because we want to extract and study information relative to these connections. In this scenario, graph databases are the ideal, they are designed and implemented to deal with connected information in a efficient way. In this talk I'll explain why NoSQL is necessary in some contexts as an alternative to traditional relational databases. How graph databases allow developers model their domains in a natural way without translating these domain models to an relational model with some artificial data like foreign keys and why is more efficient a graph database than a relational one or even a document database in a high connected environment. Then I'll explain specific characteristics of Neo4J as well as how to use Cypher the neo4j query language through python.

Francisco Fernández Castaño - Extending Python, what is the best option for me? [EuroPython 2014] [22 July 2014] Python is a great language, but there are occasions where we need access to low level operations or connect with some database driver written in C. With the FFI(Foreign function interface) we can connect Python with other languages like C, C++ and even the new Rust. There are some alternatives to achieve this goal, Native Extensions, Ctypes and CFFI. I'll compare this three ways of extending Python. ----- In this talk we will explore all the alternatives in cpython ecosystem to load external libraries. In first place we'll study the principles and how shared libraries work. After that we will look into the internals of CPython to understand how extensions work and how modules are loaded. Then we will study the main three alternatives to extend CPython: Native Extensions, Ctypes and CFFI and how to automate the process. Furthermore we will take a look to other python implementations and how we can extend it.

Benoit Chesneau - Concurrent programming with Python and my little experiment [EuroPython 2014] [25 July 2014] Concurrent programming in Python may be hard. A lot of solutions exists though. Most of them are based on an eventloop. This talk will present what I discovered and tested along the time and my little experiment in porting the Go concurrency model in Python. ----- Concurrent programming in Python may be hard. A lot of solutions exists though. Most of them are based on an eventloop. This talk will present what I discovered and tested along the time with code examples, from asyncore to asyncio, passing by gevent, eventlet, twisted and some new alternatives like evergreen or gruvi. It will also present my little experiment in porting the Go concurrency model in Python named [offset](http://github.com/benoitc/offset), how it progressed in 1 year and how it became a fully usable library . This presentation will be an update of the presentation I gave at the FOSDEM 2014. It will introduce to the concurrency concepts and how they are implemented in the different libraries.

Yves - Performance Python for Numerical Algorithms [EuroPython 2014] [23 July 2014] This talk is about several approaches to implement high performing numerical algorithms and applications in Python. It introduces into approaches like vectorization, multi-threading, parallelization (CPU/GPU), dynamic compiling, high throughput IO operations. The approach is a practical one in that every approach is illustrated by specific Python examples. ----- This talk is about several approaches to implement high performing numerical algorithms and applications in Python. It introduces into approaches like multi-threading, parallelization (CPU/GPU), dynamic compiling, high throughput IO operations. The approach is a practical one in that every approach is illustrated by specific Python examples. The talk uses, among others, the following libraries: * NumPy * numexpr * IPython.Parallel * Numba * NumbaPro * PyTables

Tarashish Mishra - Building Realtime Web Applications with WebRTC and Python [EuroPython 2014] [25 July 2014] WebRTC makes building peer to peer real time web applications easier. First, we'll discuss in short what WebRTC is, how it works. Then we will explore ways to build the signalling system of a WebRTC app using Python. ----- Introduction =========== This talk will first introduce the audience to WebRTC and then discuss about how to implement the server side logic of a WebRTC app using Python. WebRTC is a free, open project that enables web browsers with plugin-less Real-Time Communications (RTC) capabilities via simple JavaScript APIs. What makes WebRTC special is that the data travels from one client to another without going through the server. The main functions of WebRTC can be broadly categorized into three types. - Access and acquire video and audio streams - Establish a connection between peers and stream audio/video. - Communicate arbitrary data. WebRTC uses three different JavaScript APIs to perform these three functions. These APIs are: - MediaStream (aka getUserMedia) - RTCPeerConnection - RTCDataChannel MediaStream API performs the task of accessing the webcam and/or microphone of the device and acquire the video and/or audio stream from them. RTCPeerConnection API establishes connection between peers and streams audio and video data. This API also does all the encoding and decoding of audio/video data. The third API, RTCDataChannel helps to communicate arbitrary data from one client to the other. There will be short demos to demonstrate the functionalities of these APIs. Signaling and Session Control ======================== WebRTC uses RTCPeerConnection to communicate streaming data between browsers, but some sort of mechanism is needed to coordinate this communication and to send control messages. This process is known as signaling. Signaling is used to exchange three types of information. - Session control messages: to initialize or close communication and report errors. - Network configuration: to the outside world, what's my computer's IP address and port? - Media capabilities: what codecs and resolutions can be handled by my browser and the browser it wants to communicate with? This can be implemented using any appropriate two way communication channel. Implementing signaling in Python ========================== Next, we will have a look at how to implement this signaling mechanism in Python. ( Demonstration with annotated code and live application.) ### Google AppEngine and the Channel API ### Google AppEngine has a channel API which offers persistent connections between your application and Google servers, allowing your application to send messages to JavaScript clients in real time without the use of polling. We'll use this Channel API to build the signaling system of our WebRTC app on top of webapp2 and flask framework. ### Flask and gevent ### We'll implement the same signaling system again, this time on top of Flask using gevent for the persistent connection between the browser and our application. Outline of the talk =============== ### Intro (5 min) ### - Who are we? - What is WebRTC? - Functions of WebRTC. ### WebRTC APIs and Demos (3 min) ### - MediaStream (getUserMedia) API - RTCPeerConnection API - RTCDataChannel API ### Signaling in WebRTC Applications (3 min) ### - What is signaling? - Why is it needed? - How to implement it? ### Implementation of signaling (16 min) ### - Implementation using Google AppEngine and Channel API - Implementation using Flask and gevent ### Questions (3 min) ###

Florian Wilhelm - Extending Scikit-Learn with your own Regressor [EuroPython 2014] [25 July 2014] We show how to write your own robust linear estimator within the Scikit-Learn framework using as an example the Theil-Sen estimator known as "the most popular nonparametric technique for estimating a linear trend". ----- Scikit-Learn (http://scikit-learn.org/) is a well-known and popular framework for machine learning that is used by Data Scientists all over the world. We show in a practical way how you can add your own estimator following the interfaces of Scikit-Learn. First we give a small introduction to the design of Scikit-Learn and its inner workings. Then we show how easily Scikit-Learn can be extended by creating an own estimator. In order to demonstrate this, we extend Scikit-Learn by the popular and robust Theil-Sen Estimator (http://en.wikipedia.org/wiki/Theil%E2%80%93Sen_estimator) that is currently not in Scikit-Learn. We also motivate this estimator by outlining some of its superior properties compared to the ordinary least squares method (LinearRegression in Scikit-Learn).

Jim Baker - Scalable Realtime Architectures in Python [EuroPython 2014] [25 July 2014] This talk will focus on you can readily implement highly scalable and fault tolerant realtime architectures, such as dashboards, using Python and tools like Storm, Kafka, and ZooKeeper. We will focus on two related aspects: composing reliable systems using at-least-once and idempotence semantics and how to partition for locality. ----- Increasingly we are interested in implementing highly scalable and fault tolerant realtime architectures such as the following: * Realtime aggregation. This is the realtime analogue of working with batched map-reduce in systems like Hadoop. * Realtime dashboards. Continuously updated views on all your customers, systems, and the like, without breaking a sweat. * Realtime decision making. Given a set of input streams, policy on what you like to do, and models learned by machine learning, optimize a business process. One example includes autoscaling a set of servers. (We use realtime in the soft sense: systems that are continuously computing on input streams of data and make a best effort to keep up; it certainly does not imply hard realtime systems that strictly bound their computation times.) Obvious tooling for such implementations include Storm (for event processing), Kafka (for queueing), and ZooKeeper (for tracking and configuration). Such components, written respectively in Clojure (Storm), Scala (Kafka), and Java (ZooKeeper), provide the desired scalability and reliability. But what may not be so obvious at first glance is that we can work with other languages, including Python, for the application level of such architectures. (If so inclined, you can also try reimplementing such components in Python, but why not use something that's been proven to be robust?) In fact Python is likely a better language for the app level, given that it is concise, high level, dynamically typed, and has great libraries. Not to mention fun to write code in! This is especially true when we consider the types of tasks we need to write: they are very much like the data transformations and analyses we would have written of say a standard Unix pipeline. And no one is going to argue that writing such a filter in say Java is fun, concise, or even considerably faster in running time. So let's look at how you might solve such larger problems. Given that it was straightforward to solve a small problem, we might approach as follows. Simply divide up larger problems in small one. For example, perhaps work with one customer at a time. And if failure is an ever present reality, then simply ensure your code retries, just like you might have re-run your pipeline against some input files. Unfortunately both require distributed coordination at scale. And distributed coordination is challenging, especially for real systems, that will break at scale. Just putting a box in your architecture labeled **"ZooKeeper"** doesn't magically solve things, even if ZooKeeper can be a very helpful part of an actual solution. Enter the Storm framework. While Storm certainly doesn't solve all problems in this space, it can support many different types of realtime architectures and works well with Python. In particular, Storm solves two key problems for you. **Partitioning**. Storm lets you partition streams, so you can break down the size of your problem. But if the a node running your code fails, Storm will restart it. Storm also ensures such topology invariants as the number of nodes (spouts and bolts in Storm's lingo) that are running, making it very easy to recover from such failures. This is where the cleverness really begins. What can you do if you can ensure that **all the data** you need for a given continuously updated computation - what is the state of this customer's account? - can be put in **exactly one place**, then flow the supporting data through it over time? We will look at how you can readily use such locality in your own Python code. **Retries**. Storm tracks success and failure of events being processed efficiently through a batching scheme and other cleverness. Your code can then choose to retry as necessary. Although Storm also supports exactly-once event processing semantics, we will focus on the simpler model of at-least-once semantics. This means your code must tolerate retry, or in a word, is idempotent. But this is straightforward. We have often written code like the following: seen = set() for record in stream: k = uniquifier(record) if k not in seen: seen.add(k) process(record) Except of course that any such real usage has to ensure it doesn't attempt to store all observations (first, download the Internet! ;), but removes them by implementing some sort of window or uses data structures like HyperLogLog, as we will discuss.

adam - How Disqus is using Django as the basis of our Service Oriented Architecture [EuroPython 2014] [24 July 2014] Disqus maintains the largest Django app out there. And we love it! It has, however, grown rather large and unwieldy. In the last year Disqus has had an increasing number of smaller services cropping up based on several different platforms. So this talk will be about how we do continuous deployment with our emerging service-based infrastructure.

Valerio Maggio - Traversing Mazes the pythonic way and other Algorithmic Adventures [EuroPython 2014] [22 July 2014] Graphs define a powerful mental (and mathematical) model of structure, representing the building blocks of formulations and/or solutions for many hard problems. In this talk, graphs and (some of the) main graph-related algorithms will be analysed from a very pythonic angle: the Zen of Python (e.g., beautiful is better than ugly, simple is better than complex, readability counts).

Naomi Ceder - Farewell and Welcome Home: Python in Two Genders [EuroPython 2014] [24 July 2014] After half a lifetime I transitioned from male to female while staying involved in the Python community. This talk discusses that transition and explores how I found life in Python as a woman different from my former life as a man and the lessons about diversity I have learned. ----- After half a lifetime "undercover as a man" I transitioned from male to female while staying involved in the Python community. This talk discusses that transition and explores how I found life in Python as a woman different from my former life as a man and the lessons about diversity I have learned. This talk will include a brief discussion of what being transgender means, my experiences as I came to terms with it, and the losses and gains transition entailed. Early on I made the decision to be as open as possible and to stay engaged in the Python community as I transitioned and I will discuss why I made that decision and the levels of acceptance and support I encountered. Transition has been wonderfully successful, but that very transition put me in a surprisingly different world. Now being part of not one, but at least 3 groups that are minorities in the Python world gave me a very different view of a community I thought I knew, and pushed me to being an activist (or trouble maker) in spite of myself. In addition to the many positives the Python community has offered me on my journey, I will discuss the experiences that have made me understand that privilege is very much alive and well in the Python world.

Celia - 3D sensors and Python: A space odyssey [EuroPython 2014] [23 July 2014] This talk will show how to build a simple open source based NUI (Natural User Interface) game with 3D Sensors, incorporating PyOpenNI with PyGame and WebGL. OpenNI allows you operate several 3D sensors, enabling hardware independent game development (supported 3D sensors are Microsoft Kinect, PrimeSense Carmine or Asus XTion). It also runs on Linux, Mac OS X and Windows. ----- This talk will start with a brief introduction to 3D Sensors and OpenNI. Then we’ll surf into PyOpenNI, features such as the skeleton, hand and gesture tracking, RGB and depth video. Every topic will be presented with practical demos. The talk will end with a demo integrating WebGL (THREE.JS), 3D sensors, Flask and ZMQ to produce a simple fully open source based NUI game. Some simple demos of PyOpenNI and PyGame can be found at [1](http://www.youtube.com/watch?v=wI2ktioiPY8) and [2](http://youtu.be/3e8jibGUQ2Q) Attendees will not only learn about game related technologies but also about innovative ways of doing domotics, cinema & art, Interactive visualization, scientific research, educations, etc. 3D Sensors will be available for testing during the event - you can get yours for about 80 to 140 Euros (depending on the brand). Slides and demo code will be available at Github. Talk structure: * Introduction: hardware and OpenNI goodies and a tale of PCL (5’) * Hands On PyOpenNI * Normal and Depth camera - basics concepts and small demo (5’) * Skeleton - basics concepts and small demo. (5’) * Hand & gesture - basics concepts and small demo. (5’) * Final Demo * What we’re going to use? Flask, ZMQ, THREE.JS, PyOpenNI. (6’) * Q&A. (4’)

Jozef - Amanda: A New Generation of Distributed Services Framework [EuroPython 2014] [22 July 2014] Abstract To help create award winning visual effects, MPC developed a distributed service-oriented platform, Amanda. Amanda allows developers of any level to write a service that is presented to users across 8 facilities globally without them requiring any knowledge of building large concurrent systems. It allows artists and developers across different domains to work with clearly defined API's and gives the service developer control over what and how data can and should be accessed. The talk will cover how to set up such a platform from the ground up. Starting at the service level building it out with additional modules and technologies until the fully distributed system, covering topics such as concurrency, componetisation and monitoring that allow the fine tuning of setups depending on the type of work being undertaken and changing business needs.

Matt Williams - Ganga: an interface to the LHC computing grid [EuroPython 2014] [25 July 2014] Ganga is a tool, designed and used by the large particle physics experiments at CERN. Written in pure Python, it delivers a clean, usable interface to allow thousands of physicists to interact with the huge computing resources available to them. ----- [Ganga](https://cern.ch/ganga) is a tool, designed and used by the large particle physics experiments at CERN. Written in pure Python, it delivers a clean, usable interface to allow thousands of physicists to interact with the huge computing resources available to them. It provides a single platform with which data analysis tasks can be run on anything from a local machine to being distributed seamlessly to computing centres around the world. The talk will cover the problems faced by physicists when dealing with the computer infrastructure and how Ganga helps to solve this problem. It will focus on how Python has helped create such a tool through its advanced features such as metaclasses and integration into IPython.

whitone - GNU/Linux Hardware Emulation with Python [EuroPython 2014] [22 July 2014] With the kernel inotify feature, the D-Bus mocker library and the udev monitoring we try to detect the different events that occours when you're using a specific set of connected devices. Then we try to mimic these devices investigating also the kernel drivers if necessary. At the end we're ready to connect the simulation routines to our testing procedure. ----- With the kernel [inotify](http://en.wikipedia.org/wiki/Inotify "inotify") feature, the [D-Bus mocker library](https://launchpad.net/python-dbusmock "D-Bus mocker library") and the [udev monitoring](http://pyudev.readthedocs.org/en/latest/api/pyudev.html#pyudev.Monitor "udev monitoring") we try to detect the different events that occours when you're using a specific set of connected devices. Then we try to mimic these devices investigating also the kernel drivers if necessary. At the end we're ready to connect the simulation routines to our testing procedure.

Armin Rigo/Romain Guillebert - PyPy status talk (a.k.a.: no no, PyPy is not dead) [EuroPython 2014] [22 July 2014] The current status of PyPy, with a particular focus on what happened in the last two years, since the last EuroPython PyPy talk. We will give a brief overview of the current speed and the on-going development efforts on the JIT, the GC, NumPy, Python 3 compatibility, CFFI, STM... http://pypy.org/talk/ep2014-status.html ----- In this talk we will present the current status of PyPy, with a particular focus on what happened in the last two years, since the last EuroPython PyPy talk. We will give an overview of the current speed and the on-going development efforts, including but not limited to: - the status of the Just-in-Time Compiler (JIT) and PyPy performance in general; - the improvements on the Garbage Collector (GC); - the status of the NumPy and Python 3 compatibility subprojects; - CFFI, which aims to be a general C interface mechanism for both CPython and PyPy; - a quick overview of the STM (Software Transactional Memory) research project, which aims at solving the GIL problem. This is the "general PyPy status talk" that we give every year at EuroPython (except last year; hence the "no no, PyPy is not dead" part of the title of this talk).

Armin Rigo - Using All These Cores: Transactional Memory in PyPy [EuroPython 2014] [23 July 2014] PyPy, the Python implementation written in Python, experimentally supports Transactional Memory (TM). The strength of TM is to enable a novel use of multithreading, inheritently safe, and not limited to special use cases like other approaches. This talk will focus on how it works under the hood. http://pypy.org/talk/ep2014-stm.html ----- PyPy is a fast alternative Python implementation. Software Transactional Memory (STM) is a current academic research topic. Put the two together --brew for a couple of years-- and we get a version of PyPy that runs on multiple cores, without the infamous Global Interpreter Lock (GIL). The current research is based on a recent new insight that promises to give really good performance. The speed of STM is generally measured by two factors: the ability to scale with the number of CPUs, and the amount of overhead when compared with other approaches in a single CPU (in this case, with the regular PyPy with the GIL). Scaling is not really a problem here, but single-CPU performance is --or used to be. This new approach gives a single-threaded overhead that should be very low, maybe 20%, which would definitely be news for STM systems. Right now (February 2014) we are still implementing it, so we cannot give final numbers yet, but early results on a small interpreter for a custom language are around 15%. This looks like a deal-changer for STM. In the talk, I will describe our progress, hopefully along with real numbers and demos. I will then dive under the hood of PyPy to give an idea about how it works. I will conclude with a picture of how the future of multi-threaded programming might looks like, for high-level languages like Python. I will also mention CPython: how hard (or not) it would be to change the CPython source code to use the same approach.

Anselm Kruis - Post-Mortem Debugging with Heap-Dumps [EuroPython 2014] [25 July 2014] UNIX core-dumps, Windows minidumps or Java heap-dumps are well established technologies for post-mortem defect analysis. I'll present a similar technology for Python. An improved pickling mechanism makes it possible to serialise the state of a Python program for subsequent analysis with a conventional Python-debugger. ----- Post-Mortem Debugging with Heap-Dumps === UNIX core-dumps, Windows minidumps and analogous solutions of other operating systems are well established technologies for post-mortem defect analysis of native-code processes. In principle those dumps can be used to analyse „interpreted“ programs running within a native-code interpreter-process. However in practise this approach is tedious and not always successful \[1\]. Therefore operating system independent dump methods were developed for some „interpreted“ languages \[2\]. A prominent example are Java heap dumps \[3\]. Unfortunately up to now there was no practically usable dump-method for Python. Various attempts were made to utilise OS-level dump methods \[4, 5\]. In 2012 Eli Finer published the Python module *pydump* \[6\]. This module pickles the traceback of an exception and subsequently uses the pdb debugger to analyse the unpickled traceback. Unfortunately *pydump* fails on PicklingErrors. In my talk I'll present the Python package [*pyheapdump*](https://pypi.python.org/pypi/pyheapdump). It has the same operation principle as Eli's *pydump*, but is an independent implementation. *pyheapdump* uses an extended pickler ([sPickle](https://pypi.python.org/pypi/sPickle)) to serialise all relevant objects of a Python process to a file. Later on a fault tolerant unpickler recreates the objects and a common Python debugger can be used to analyse the dump. The pickler extensions make it possible to: * pickle and unpickle many commonly not pickleable objects [7]. * replace the remaining not pickleable objects by surrogate objects so that the resulting object graph is almost isomorphic to the original object graph. Which objects are relevant? In its default operation mode *pyheapdump* uses the frame-stacks of all threads as start point for pickling. Following the usual rules for pickling the dump includes all local variables and all objects reachable from a local variable and so on. That is usually enough for a successful defect analysis. Compared with other Python post-mortem debugging methods *pyheapdump* has several advantages: * It is a pure Python solution and independent from the operation system. * Creation of the pyheapdump and fault analysis can be performed different computers. * It is not obstructive. It does not modify / monkey-patch or disturb the dumped process in any way, with the exception of loading additional modules. * If used with the Pydev-debugger, it supports multi-threaded applications. * If used with the Pydev-debugger and Stackless Python, it supports tasklets. The implementation of *pyheapdump* is fairly small, because it draws most of its functionality from the underlying sPickle package and from the new Stackless-Support \[8\] of the Pydev-Debugger. Therefore it is - despite of its short history - already a useful piece of software. Outline of the talk --- 1. Introduction to the problem 2. Previous works 3. The concept of *pyheapdump* 4. Live demonstration 5. Open problems and further development 6. Questions and Answers References --- 1. Andraz Tori, Python, 2011-01-16: *gdb and a very large core dump*, blog at <http://www.zemanta.com/blog/python-gdb-large-core-dump/> 2. David Pacheco, ACM Queue - Programming Languages Volume 9 Issue 10, October 2011: *Postmortem Debugging in Dynamic Environments*, PDF <http://dl.acm.org/ft_gateway.cfm?id=2039361&ftid=1050739&dwn=1&CFID=290171300&CFTOKEN=95099236> 3. Chris Bailey, Andrew Johnson, Kevin Grigorenko, IBM developerWorks, 2011-03-15: *Debugging from dumps - Diagnose more than memory leaks with Memory Analyzer*, PDF <http://www.ibm.com/developerworks/library/j-memoryanalyzer/j-memoryanalyzer-pdf.pdf> 4. Brian Curtin, 2011-09-29: *minidumper - Python crash dumps on Windows*, blog at <http://blog.briancurtin.com/posts/20110929minidumper-python-crash-dumps-on-windows.html> 5. David Malcolm, Fedora Feature, 2010-04-06: *Easier Python Debugging* at <http://fedoraproject.org/wiki/Features/EasierPythonDebugging> 6. Eli Finer, Github-Project, 2012: *pydump* at <https://github.com/gooli/pydump> 7. Anselm Kruis, EuroPython 2011: *Advanced Pickling with Stackless Python and sPickle*, archived talk at <https://ep2013.europython.eu/conference/talks/advanced-pickling-with-stackless-python-and-spickle> 8. Fabio Zadrozny, 2013-12-12: *PyDev 3.1.0 released* blog at <http://pydev.blogspot.de/2013/12/pydev-310-released.html>

Adriana Vasiu - Cutting-edge APIs using hypermedia at BSkyB [EuroPython 2014] [22 July 2014] In this talk I will explain what hypermedia enabled API means, I will give an example of such an API and I will take you through the implementation details and the usage of flask, dougrain and HAL in this context. Also, I will present a brief comparison with an API that is not hypermedia enabled and take you through the advantages of using the hypermedia approach. ----- In the technology community at the moment there is a lot of talk about hypermedia enabled APIs and Web as an Architecture model. More and more applications nowadays try to adopt the loosely coupled and distributed web like architecture by using hypermedia as an engine of the application state. In Sky we are successfully implementing this approach for some of our components, and we’ve learnt that the major benefit for us is the scalability that it offers: as an increasingly expanding business with a constantly growing product portfolio, scalability of all our systems is crucial. In this talk I will share some of the things we learnt, I will explain what hypermedia enabled API means, I will give an example of such an API and I will take you through the implementation details and the usage of flask, dougrain and HAL in this context. Also, I will present a brief comparison with an API that is not hypermedia enabled and take you through the advantages of using the hypermedia approach.

Peter Hoffmann - log everything with logstash and elasticsearch [EuroPython 2014] [22 July 2014] When your application grows beyond one machine you need a central space to log, monitor and analyze what is going on. Logstash and elasticsearch let you store your logs in a structured way. Kibana is a web fronted to search and aggregate your logs. ----- The talk will give an overview on how to add centralized, structured logging to a python application running on multiple servers. It will focus on useful patterns and show the benefits from structured logging.

Francesc Alted - Out-of-Core Columnar Datasets [EuroPython 2014] [25 July 2014] Tables are a very handy data structure to store datasets to perform data analysis (filters, groupings, sortings, alignments...). But it turns out that how the tables are actually implemented makes a large impact on how they perform. Learn what you can expect from the current tabular offerings in the Python ecosystem. ----- It is a fact: we just entered in the Big Data era. More sensors, more computers, and being more evenly distributed throughout space and time than ever, are forcing data analyists to navigate through oceans of data before getting insights on what this data means. Tables are a very handy and spreadly used data structure to store datasets so as to perform data analysis (filters, groupings, sortings, alignments...). However, the actual table implementation, and especially, whether data in tables is stored row-wise or column-wise, whether the data is chunked or sequential, whether data is compressed or not, among other factors, can make a lot of difference depending on the analytic operations to be done. My talk will provide an overview of different libraries/systems in the Python ecosystem that are designed to cope with tabular data, and how the different implementations perform for different operations. The libraries or systems discussed are designed to operate either with on-disk data ([PyTables] [1], [relational databases] [2], [BLZ] [3], [Blaze] [4]...) as well as in-memory data containers ([NumPy] [5], [DyND] [6], [Pandas] [7], [BLZ] [3], [Blaze] [4]...). A special emphasis will be put in the on-disk (also called out-of-core) databases, which are the most commonly used ones for handling extremely large tables. The hope is that, after this lecture, the audience will get a better insight and a more informed opinion on the different solutions for handling tabular data in the Python world, and most especially, which ones adapts better to their needs. [1]: http://www.pytables.org [2]: http://en.wikipedia.org/wiki/Relational_database [3]: http://blz.pydata.org [4]: http://blaze.pydata.org [5]: http://www.numpy.org/ [6]: https://github.com/ContinuumIO/dynd-python [7]: http://pandas.pydata.org/

Schlomo Schapiro - DevOps Risk Mitigation: Test Driven Infrastructure [EuroPython 2014] [23 July 2014] The (perceived) risk of the DevOps is that too many people get the right to "break" the platform. Test Driven Infrastructure is about adapting proven ideas from our developer colleagues to the development and operations of Infrastructure services like virtualization, OS provisioning, postfix configuration, httpd configuration, ssh tuning, SAN LUN mounting and others. This talk shows how ImmobilienScout24 utilizes more and more test driven development in IT operations to increase quality and to mitigate the risk of opening up the infrastructure developmen to all developers. ----- Common wisdom has it that the test effort should be related to the risk of a change. However, the reality is different: Developers build elaborate automated test chains to test every single commit of their application. Admins regularly “test” changes on the live platform in production. But which change carries a higher risk of taking the live platform down? What about the software that runs at the “lower levels” of your platform, e.g. systems automation, provisioning, proxy configuration, mail server configuration, database systems etc. An outage of any of those systems can have a financial impact that is as severe as a bug in the “main” software! One of the biggest learnings that any Ops person can learn from a Dev person is Test Driven Development. Easy to say - difficult to apply is my personal experience with the TDD challenge. This talk throws some light on recent developments at ImmobilienScout24 that help us to develop the core of our infrastructure services with a test driven approach: * How to do unit tests, integration tests and systems tests for infrastructure services? * How to automatically verify Proxy, DNS, Postfix configurations before deploying them on live servers? * How to test “dangerous” services like our PXE boot environment or the automated SAN mounting scripts? * How to add a little bit of test coverage to everything we do. * Test Driven: First write a failing test and then the code that fixes it. The tools that we use are Bash, Python, Unit Test frameworks and Teamcity for build and test automation. See http://blog.schlomo.schapiro.org/2013/12/test-driven-infrastructure.html for more about this topic.

ssc - Event discrete simulation with SimPy [EuroPython 2014] [25 July 2014] Often, experiments with real world systems are high-risk, accompanied by high costs or not even possible at all. That’s when simulations come into play. This talk will give a brief introduction into the topic of simulation. By means of simple examples, it will demonstrate how you can use SimPy to implement event-discrete simulations and which features SimPy offers to help you doing that. ----- Simulation is important for the analysis of complex systems or the analysis of the impact of certain actions on that systems. They are especially useful if the actions are potentially harmful or expensive. Simulation is used in various natural scientific and economic areas, e.g., for the modeling and study of biological or physical systems, for resource scheduling and optimization or at the research for the integration of renewable energies into the power grid (my personal background). The simulated time can thereby be seen as continuous or discrete (discrete time or discrete event). In this talk, I want to show why Python is a good choice for implementing simulation models and how SimPy can help here. Structure of the talk (20min talking + 5min discussion + 5min buffer): - Why simulation? (5min) - History of SimPy (3min) - How does SimPy work? (9min) - Conclusion (3min) In the introduction, I’ll briefly explain what simulation is and motivate, why it is a useful tool. The main part will consist of an introduction and demonstration of SimPy. Since SimPy is now more then ten years old, I’ll first give a quick overview about its history and development. Afterwards, I’ll explain SimPy’s concepts and features by means of simple examples. In the conclusion, I’ll give a short outlook on the future development of SimPy. The main goal of this talk is to create awareness that simulation is a powerful tool in a lot of domains and to give the audience enough information to ease their first steps.

Matt - Full Stack Python [EuroPython 2014] [22 July 2014] There has been a lot of noise about being a "full stack developer" recently. What does the full web stack look like for Python and how do you go about learning each piece? This talk will guide you up the layers from the server that handles the web request through the JavaScript that executes on a user's browser. ----- This talk distills information from the open source guide [Full Stack Python](http://www.fullstackpython.com/) I wrote into a 30 minute talk on web stack layers. An approximate timeline for this talk would be: * 5 min: intro story * 5 min: what the web developers need to know about virtual servers, web servers, and WSGI servers * 5 min: what do web frameworks provide? * 5 min: what are the most important parts of your web application to analyze and monitor? * 5 min: static files and execution on the user's browser * 5 min: concluding story and resources to learn more This is a high level overview intended for developers who are new to Python web development and need to understand what the web stack layers are and how they fit together.

Stefanie Lück - RISCy Business: Development of a RNAi design and off-target prediction software [EuroPython 2014] [24 July 2014] RNA interference (RNAi) is a biological mechanism for targeted inhibition of gene expression. It has also been used routinely to discover genes involved in the interaction of plants with pathogenic fungi. To minimize the miss-targeting of unrelated genes and to maximize the RNAi efficiency, we have developed a PyQt based cross- platform software tool called “si-Fi”. Our aim of the talk is to show that also hobby programmers can use Python in a very useful way.

Tom Christie - Documenting your project with MkDocs. [EuroPython 2014] [22 July 2014] MkDocs is a new tool for creating documentation from Markdown. The talk will cover: How to write, theme and publish your documentation. The background and motivation for MkDocs. Choosing between MkDocs or Sphinx. ----- This talk will be a practical introduction to MkDocs, a new tool for creating documentation from Markdown: * The background behind MkDocs and the motivation for creating a new documentation tool. * Comparing against Sphinx - what benefits each tool provides. * Getting starting with MkDocs - how to write, theme and publish your documentation. * Under the covers - how MkDocs works, and some asides on a couple of the neat Python libraries that it uses.

Alex Brasetvik - Elasticsearch from the bottom up [EuroPython 2014] [24 July 2014] This talk will teach you about Elasticsearch and Lucene's architecture. The key data structure in search is the powerful inverted index, which is actually simple to understand. We start there, then ascend through abstraction layers to get an overview of how a distributed search cluster processes searches and changes. ----- ## Who I am and motivation I work with hosted Elasticsearch and have interacted with lots of developers. We see what many struggle with. Some relevant theory helps a lot. What follows has already lead to many "Aha!"-moments and developers piecing things together herself. ## The inverted index The most important index structure is actually very simple. It is essentially a sorted dictionary of terms, with a list of postings per term. We show three simple sample documents and the resulting inverted index. ## The index term The index term is the "unit of search", and the terms we make decide how we can search. With the inverted index and its sorted dictionary, we can quickly search for terms given their prefix. ## Importance of text analysis Thus, we need to transform our search problems into string prefix problems. This is done with text analysis, which is the process of making of index terms. It is highly important when implementing search. ## Building indexes The way indexes are built must balance how compact an index is, how easily we can search in it, how fast we can index documents - and the time it takes for changes to be visible. Lucene, and thus Elasticsearch, builds them in segments. ## Index segments A Lucene index consists of index segments, i.e. immutable mini-indexes. A search on an index is done by doing the search on all segments and merging the results. Segments are immutable: This enables important compression techniques. Deletes are not immediate, just a marker. Segments are occasionally merged to larger segments. Then documents are finally deleted. New segments are made by buffering changes in memory, and written when flushing happens. Flushes are largely caused by refreshing every second, due to real time needs. ## Caches Caches like filter- and field caches are managed per segment. They are essential for performance. Immutable segments make for simple reasoning about caches. New segments only cause partial cache invalidations. ## Elasticsearch indexes Much like a Lucene index is made up of many segments, an Elasticsearch index is made up of many Lucene indexes. Two Elasticsearch indexes with 1 shard is essentially the same as one Elasticsearch index with 2 shards. Search all shards and merge. Much like segments, but this time possibly across machines. Shard / Index routing enables various partitioning strategies. Simpler than it sounds, so one important example: Essential for time based data, like logs: can efficiently skip searching entire indexes - and roll out old data by deleting the entire index. ## Common pitfalls We must design our indexing for how we search - not the searches for how things are indexed. Be careful with wildcards and regexes. Since segments are immutable, deleting documents is expensive while deleting an entire index is cheap. Updating documents is essentially a delete and re-index. Heavy updating might cause problems. Have enough memory and then some. Elasticsearch is very reliant on its caches. ## Summary We've seen how index structures are used, and why proper text processing is essential for performant searches. Also, you now know what index segments are, and how they affect both indexing and searching strategies. ## Questions

Domen Kožar - Rethinking packaging, development and deployment [EuroPython 2014] [22 July 2014] In Python, we're trying to solve packaging problems in our own domain, but maybe someone else already solved most our problems. In the talk I'll show how I develop and deploy Python projects that can be easily mixed with non-Python dependencies. http://nixos.org/nix/ will be demonstrated to replace technologies in our stack: pip, virtualenv, buildout, ansible, jenkins. ----- Python is often mixed with other languages in development stack, nowadays it's hard to escape any JavaScript dependencies. If you add some C dependencies such as GStreamer to the stack, packaging becomes a burden. While tweaking our packaging infrastructure will make things better, it's hard to fix fundamental problem of packaging with current ad-hoc solutions in Python domain. Using Nix (http://nixos.org/nix/) for about a year gave me an insight that solving packaging problem at operating system level (bottom-up) is a better approach. For example, wouldn't it be cool to have "virtualenv" implemented inside your package manager, so you could isolate also non-Python dependencies and not just Python packages for your project and not worry if system was updated? We'll also show what benefits do we get by using the same tool for development and deployment and how little we have to do to deploy our application. To see how Haskell community is touching the same subject, see blog post http://ocharles.org.uk/blog/posts/2014-02-04-how-i-develop-with-nixos.html

Felix Wick/Florian Wilhelm - How to Setup a new Python Project [EuroPython 2014] [23 July 2014] Setting up a new Python project from scratch can be quite hard. How to structure your files and directories. Where should my packages, modules, documentation and unit tests go? How do I configure setup.py, Sphinx and so on? We provide proven answers! ----- Whenever a Python beginner starts with its own project he or she is confronted with the same technical questions. Questions about a well thought out directory structure to hold all the files. How setup.py needs to be configured and even what it is capable of like specifying entry_points and other goodies. We show from the experience of our yearslong work with Python how to structure your Python project in terms of folders, files, modules and packages. How to configure setup.py to specify your requirements, to use it with nosetests, with Sphinx and so on. We also elaborate on the usage of Git and Versioneer (https://github.com/warner/python-versioneer) to help you version your package.

Prashant Agrawal - Jigna: a seamless Python-JS bridge to create rich HTML UIs for Python apps [EuroPython 2014] [22 July 2014] Jigna aims to provide an easy way to create rich user interfaces for Python applications using web technologies like HTML, CSS and Javascript, as opposed to widget based toolkits like Qt/wx or native toolkits. It provides a seamless two-way data binding between the Python model and the HTML view by creating a Python-JS communication bridge. This ensures that the view is always live as it can automatically update itself when the model changes, and update the model when user actions take place on the UI. The Jigna view can be rendered in an in-process Qt widget or over the web in a browser.

Mark Smith - Writing Awesome Command-Line Programs in Python [EuroPython 2014] [24 July 2014] Command-Line programs can have a lot to them - usually more than you think, yet often suffer from a lack of thought. This is a tour through how to structure your code, tools in the standard library and some 3rd party libraries. Take your command-line programs to the next level! ----- Python is a great language for writing command-line tools - which is why so much of Linux is secretly written in Python these days. Unfortunately, what starts as a simple script can quickly get out of hand as more features are added and more people start using it! The talk will consist of a tour through various useful libraries and practical code showing how each can be used, and include advice on how to best structure simple and complex command-line tools. Things to consider when writing command-line apps: * Single-file vs Multiple-file * Standard library only vs. 3rd party requirements * Installation - setup.py vs. native packaging The different parts of a command-line program: * Option Parsing: * Libraries: getopt, optparse, argparse, docopt * Sub-commands * Configuration: * Formats: Ini file, JSON, YAML * Where should it be stored (cross-platform); * Having multiple configuration files, and allowing user config to override global config * Output: * Colour - colorama * Formatting output for the user * Formatting output for other programs * How do you know when your output is being piped to another program? * Managing logging and verbosity * Managing streamed input * Exit values: What are the conventions? * Interactive apps - REPL * Structuring a bunch of programs/commands around a shared codebase. * Command-line frameworks: clint, compago & cliff * Testing command-line apps * Writing command-line tools in Python 3 vs Python 2

Julie Pichon - I want to help! How to make your first contribution to open-source. [EuroPython 2014] [23 July 2014] Do you like open-source? Would you like to give back somehow but are not sure what to do or where to start? Together we will look at the usual workflow for making any kind of contribution, using a real patch as an example. ----- This talk aims to show at a high-level what is the process for contributing to most open-source projects. We will go from discovering a project to how to find the contributor guidelines, prepare your contribution for submission and what happens next. The general principles will be illustrated with an example from the speaker's first contribution to OpenStack. The target audience for the talk is people who have never contributed to open-source, though they would like to. Although the example will be a code contribution, the process as described applies to all kinds of contributions.

Michael König - Embedding Python: Charming the Snake with C++ [EuroPython 2014] [23 July 2014] At the example of our in-house distributed scheduling system, we discuss the challenges of embedding the Python interpreter in a C++ program. Besides the actual integration of the interpreter, efficient data exchange between both languages is discussed. In addition, this presentation demonstrates how higher-level abstractions further diminish the language barrier. ----- Python with its huge standard library and sophisticated packages developed by its thriving community has become an incredibly useful tool for data scientists. At Blue Yonder, we value Python for the ease with which we can access and combine machine learning algorithms to build accurate prediction models. To get the most business value out of the use of Python, we strive to rid our model developers from all burdens outside their core expertise, i.e., developing statistical models. To leverage our existing infrastructure, essentially a distributed scheduling system written in C++, we decided to embed a Python interpreter in our application. The goal was to let developers use the language best suited for their problem, and to let them incorporate code created by others even if it is not written in the same language. In this presentation, I will talk about a few obstacles which we had to overcome in integrating the (C)Python interpreter in our C++ program, e.g., clean resource management, error handling, and broken features in the interpreter's API. I will show how we employed features from the [Boost Python C++ library](http://www.boost.org/doc/libs/1_55_0/libs/python/) not only for simple data exchange, but also for more powerful concepts such as data sources. Finally, I will demonstrate how C++ objects can be used to seamlessly interact with Python, for example to use Python's logging package as usual while the actual logging is handled by our C++ application. With this combination of both worlds, we achieved a desirable mix of virtues: safe, reliable operations; good run-time performance; fast development; and highly expressive, unit testable core domain logic.

juliass - Multiplatform binary packaging and distribution of your client apps [EuroPython 2014] [24 July 2014] Distributing your python app to clients it’s a common task that can become hard when “stand alone” and “obfuscated code” come as requirements. Common answers in forums are on the lines of “Python is not the language you’re looking for” or “What are you trying to hide?” but another answer is possible.

Deni Bertovic - Supercharge your development environment using Docker [EuroPython 2014] [23 July 2014] These days applications are getting more and more complex. It's becoming quite difficult to keep track of all the different components an application needs to function (a database, an AMQP, a web server, a document store...). It keeps getting harder and harder to setup new development environments and to bring new developers into the team. Stuff works on one dev machine but doesn't on others? Code breaks often when deployed to production even though all tests were passing and it worked on the dev machine? The idea of this talk is to convey how important it is that we have our development environment as close to production as possible. That means setting up all those various services on your laptop/workstation. ----- In this talk I am going to show how to utilize light weight lxc containers using docker, and make your development process much more straightforward. How to share container images among your development team and be sure that everyone is running the exact same stack. Do all this without hogging too many resources, without the need for complex provisioning scripts and management systems. And above all else, how to do it fast! Rough Guidelines: 1. Describe what is LXC (Linux containers) 2. Benefits of using containers instead of traditional VM's 2. Explain where Docker comes in 3. Show how to build simple containers using Dockefile syntax 4. What are container images and how to share them 5. How to share private container images 6. Tips and tricks on how to automate

Chesnay Schepler - Big Data Analytics with Python using Stratosphere [EuroPython 2014] [25 July 2014] Stratosphere is a distributed platform for advanced big data analytics. It features a rich set of operators, advanced, iterative data flows, an efficient runtime, and automatic program optimization. We present Stratophere's new Python programming interface. It allows Python developers to easily get their hands on Big Data. ----- [Stratosphere](http://stratosphere.eu/) is implemented in Java. In 2013 we introduced support for writing Stratosphere programs in Scala. Since Scala also runs in the Java JVM the language integration was easy for Scala. In late 2013, we started to develop a generic language binding framework for Stratosphere to support non-JVM languages such as Python, JavaScript, Ruby but also compiled languages such as C++. The language binding framework uses [Google’s Protocol Buffers](https://code.google.com/p/protobuf/) for efficient data serialization and transportation between the languages. Since many “Data Scientists” and machine learning experts are using Python on a daily basis, we decided to use Python as the reference implementation for Stratosphere’s language binding feature. Our talk at the EuroPython 2014 will present how Python developers can leverage the Stratosphere Platform to solve their big data problems. We introduce the most important concepts of Stratosphere such as the operators, connectors to data sources, data flows, the compiler, iterative algorithms and more. Stratosphere is a mature, next generation big-data analytics platform developed by a vibrant [open-source community](https://github.com/stratosphere/stratosphere). The system is available under the Apache 2.0 license. The project started in 2009 as a joint research project of multiple universities in the Berlin area (Technische Universität, Humboldt Universität and Hasso-Plattner Institut). Nowadays it is an award winning system that has gained worldwide attention in both research and industry. A note to the program committee: As mentioned, the development of the Python language binding of Stratosphere has started a few months ago, therefore, the code is not yet in the main development branch. However, we are already able to execute the “Hello World” of big data, the “Word Count” example using the Python interface. See this example in the development branch: https://github.com/filiphaase/stratosphere/blob/langbinding/stratosphere-addons/stratosphere-language-binding/src/main/python/eu/stratosphere/language/binding/wordcountexample/WordCountPlan.py Please contact us if you have any questions!

Piotr Przymus - Everything You Always Wanted to Know About Memory in Python But Were Afraid to Ask [EuroPython 2014] [25 July 2014] Have you ever wondered what happens to all the precious RAM after running your 'simple' CPython code? Prepare yourself for a short introduction to CPython memory management! This presentation will try to answer some memory related questions you always wondered about. It will also discuss basic memory profiling tools and techniques. ----- This talk will cover basics of CPython memory usage. It will start with basics like objects and data structures representation. Then advanced memory management aspects, such as sharing, segmentation, preallocation or caching, will be discussed. Finally, memory profiling tools will be presented.

Nicholas Tollervey/holger krekel - The Return of "The Return of Peer to Peer Computing". [EuroPython 2014] [24 July 2014] At last year's Europython Holger Krekel gave a keynote called "The Return of Peer to Peer Computing". He described how developers, in light of the Snowden surveillance revelations, ought to learn about and build decentralized peer-to-peer systems with strong cryptography. This talk introduces, describes and demonstrates ideas, concepts and code that a group of Pythonistas have been working on since Holger's keynote. We asked ourselves two questions: what are the fundamental elements / abstractions of a peer-to-peer application and, given a reasonable answer to the first question, what can we build? We will present work done so far, discuss the sorts of application that might be written and explore how peer-to-peer technology could be both attractive and viable from an economic point of view. ----- This talk introduces, describes and demonstrates concepts and code created during sprints and via online collaboration by a distributed group of Pythonistas under the working title p4p2p (http://p4p2p.net). We asked ourselves, as frameworks such as Zope/Plone, Django, Pyramid or Flask are to web development what would the equivalent sort of framework look like for peer-to-peer application development? We've tackled several different technical issues: remote execution of code among peers, distributed hash tables as a mechanism for peer discovery and data storage, various cryptographic requirements and the nuts and bolts of punching holes in firewalls. Work is ongoing (we have another sprint at the end of March) and the final content of the talk will depend on progress made. However, we expect to touch upon the following (subject to the caveat above): * What is the problem we're trying to solve? * Why P2P? * The story of how we ended up asking the questions outlined in the abstract. * What we've done to address these questions. * An exploration of the sorts of application that could be built using P2P. * A call for helpers and collaboration. Happy to answer any questions!

Austin Bingham - Python refactoring with Rope and Traad [EuroPython 2014] [23 July 2014] Rope is a powerful Python refactoring library. Traad (Norwegian for “thread”) is a tool which makes it simpler to integrate rope into IDEs via a simple HTTP API. In this session we’ll look at how traad and rope work together and how traad integrates with at least one popular editor. ----- Python is a modern, dynamic language which is growing in popularity, but tool support for it is sometime lacking or only available in specific environments. For refactoring and other common IDE functions, however, the powerful open-source rope library provides a set of tools which are designed to be integrated into almost any programming environment. Rope supports most common refactorings, such as renaming and method extraction, but also more Python-specific refactorings, such as import organization. Rope’s underlying code analysis engine also allows it to do things like locating method definitions and generating auto-completion suggestions. While rope is designed to be used from many environments, it’s not always easy or ideal to integrate rope directly into other programs. Traad (Norwegian for “thread”) is another open-source project that addresses this problem by wrapping rope into a simple client-server model so that client programs (IDEs, editors, etc.) can perform refactorings without needing to embed rope directly. This simplifies dependencies, makes clients more robust in the face of errors, eases traad client development, and even allows clients to do things like switch between Python 2 and 3 refactoring in the same session. In this session we’ll look at how rope operates, and we’ll see how traad wraps it to provide an easier integration interface. The audience will get enough information to start using rope themselves, either directly or via traad, and they’ll see how to use traad for integrating rope into their own environments. More generally, we’ll look at why client-server refactoring tools might be preferable to the more standard approach of direct embedding.

Nicola Iarocci - Eve - REST APIs for Humans™ [EuroPython 2014] [24 July 2014] Powered by Flask, Redis, MongoDB and good intentions the Eve REST API framework allows to effortlessly build and deploy highly customizable, fully featured RESTful Web Services. The talk will introduce the project and its community, recount why and how it's being developed, and show the road ahead. ----- Nowadays everyone has data stored somewhere and needs to expose it through a Web API, possibly a RESTful one. [Eve](http://python-eve.org) is the BSD-licensed, Flask-powered RESTful application and framework that allows to effortlessly build and deploy highly customizable, fully freatured RESTful Web Services. Eve features a robust, feature rich, REST-centered API implementation. MongoDB support comes out of the box and community-driven efforts to deliver ElasticSearch and SQLAlchemy data layers are ongoing. Eve approach is such that you only need to configure your API settings and behaviour, plug in your datasource, and you’re good to go. Features such as Pagination, Sorting, Conditional Requests, Concurrency Control, Validation, HATEOAS, JSON and XML rendering, Projections, Customisable Endpoints, Rate Limiting are all included. Advanced features such as custom Authentication and Authorisation, Custom Validation, Embedded Resource Serialisation are also easily available. In my talk I will introduce the project and its community, recount why and how it's being developed, show the source code, illustrate key concepts and show the road ahead.

Erik van Zijst - The inner guts of Bitbucket [EuroPython 2014] [24 July 2014] Today Bitbucket is more than 30 times bigger than at the time of acquisition almost 4 years ago and serves repositories to over a million developers. This talk lays out its current architecture in great detail, from Gunicorn and Django to Celery and HA-Proxy to NFS. ----- This talk is about Bitbucket's architecture. Leaving no stone unturned, I'll be covering the entire infrastructure. Every component, from web servers to message brokers and load balancing to managing hundreds of terabytes of data. Since its inception in 2008, Bitbucket has grown from a standard, modest Django app into a large, complex stack that while still based around Django, has expanded into many more components. Today Bitbucket is more than 30 times bigger than at the time of acquisition almost 4 years ago and serves Git and Mercurial repos to over a million users and growing faster now than ever before. Our current architecture and infrastructure was shaped by rapid growth and has resulted in a large, mostly horizontally scalable system. What has not changed is that it's still nearly all Python based and could serve as inspiration or validation for other community members responsible for rapidly scaling their apps. This talk will layout the entire architecture and motivate our technology choices. From our Gunicorn to Celery and HA-Proxy to NFS.

Dmitry Trofimov - Python Debugger Uncovered [EuroPython 2014] [24 July 2014] This talk will explain how to implement a debugger for Python. We'll start with setting a simple trace function, then look how it is implemented in modern IDEs like PyCharm and Pydev. Then we go further in the details and uncover the tricks used to implement some cool features like exception handling and multiprocess debugging. ----- Presentation describes how to implement debugger for Python and has 4 parts: * Tracing Python code Explains how to use trace function * Debugger Architecture Explains which parts consists of a modern full-fledged debugger. * A Bit of Details Explains how to make code to work for all python versions and implementations, survive gevent monkey-patching etc. * Cool Features Explains how to implement exception handling and multiprocess debugging

Roberto Polli - Statistics 101 for System Administrators [EuroPython 2014] [22 July 2014] Python allows every sysadmin to run (and learn) basic statistics on system data, replacing sed, awk, bc and gnuplot with an unique, reusable and interactive framework. The talk is a case study where python allowed us to highlight some network performance points in minutes using itertools, scipy and matplotlib. The presentation includes code snippets and a brief plot discussion. ----- #Statistics 101 for System Administrators ## Agenda * A latency issue * Data distribution * 30 seconds correlation with pearsonr * Combinating data * Plotting and the power of color ## An use case - Network latency issues - Correlate latency with other events ## First statistics - we created our parsing library - [using various recipes](http://chimera.labs.oreilly.com/books/1230000000393/ch06.html) - Having the data in a dict like > table = { > 'time': [ 1,2,3, ..], > 'elapsed': [ 0.12, 12.43, ..], > 'error': [ 2, 0, ..], > 'size': [123,3223, ..], > 'peers': [2313, 2303, ..], - It's easy to get max, min and standard deviation > print [k, max(v), min(v), stats.mean(v) ] for k,v in table.items() ] ## Distribution - A distribution shows event frequency > from matplotlib import pyplot > pyplot.hist(table['elapsed']) - Time and Size distributions ## (Linear) Correlation - What's correlation - What's not correlation - pearsonr and probability - catch for linear correlation > from scipy.stats.stats import pearsonr > a, b = range(0,10), range(0,20, 2) > c = [randint(0,10) for x in a] > pearsonr(a, b), pearsonr(a,c) > (1.0, 0.0), (0.43, 0.2) ## Combinations - using itertools.combinations - netfishing correlation >from itertools import combination >for f1, f2 in combinations(table, 2): > r, p_value = pearsonr(table[f1], table[f2]) > print("the correlation between %s and %s is: %s" % (f1, f2, r)) > print("the probability of a given distribution (see manual) is: %s" % p_value) ## Plot always - pearsonr finds *only* linear correlation - our eyes work better :P - so...plot always! - color is the 3d dimension of a plot! > from pyplot import scatter, title, xlabel, ylabel, legend > from pyplot import savefig, close as closefig > > for f1, f2 in combinations(table, 2): > scatter(table[f1], table[2], label="%s_%s" % (f1,f2)) > # add legend and other labels > r, p = pearsonr(table[f1], table[f2]) > title("Correlation: %s v %s, %s" % (f1, f2, r)) > xlabel(f1), ylabel(f2) > legend(loc='upper left') # show the legend in a suitable corner > savefig(f1 + "_" + f2 + ".png") > closefig() ## Wrap Up! - do not use pearsonr to *exclude* relation between events - plots may serve better - scatter plot can show a system thruput and exclude correlation between fields A and fields B - continue collecting results

Fredrik Håård - Jython in practice [EuroPython 2014] [24 July 2014] A lot of people have heard of Jython, some have tried it, but it seems few have actually deployed it in a corporate environment. In this talk I'll share my experiences in using Jython as a testbed for Java applications, for rapid prototyping in Java desktop and web environments, and for embedding scripting capabilities in Java products. ----- Not everyone gets paid to work with Python all the time, but if you find yourself in a Java project, there are good chances you could benefit from Python without throwing out the Java stack. Using Jython, you can do rapid prototyping without the long edit-compile-test cycles normally associated with large Java projects, whether on the web or the desktop, and when testing an application might become a nightmare of scaffolding in Java, a little Jython may be just what you need to be able to run your tests smoothly. At the end of this talk, I will put on my politician´s hat and bring up the best - and worst - arguments to use to get permission to use Jython in a corporate environment.

M.Yasoob Khalid - Web Scraping in Python 101 [EuroPython 2014] [22 July 2014] This talk is about web scraping in Python, why web scraping is useful and what Python libraries are available to help you. I will also look into proprietary alternatives and will discuss how they work and why they are not useful. I will show you different libraries used in web scraping and some example code so that you can choose your own personal favourite. I will also tell why writing your own scrapper in scrapy allows you to have more control over the scraping process. ----- Who am I ? ========= * a programmer * a high school student * a blogger * Pythonista * and tea lover - Creator of freepythontips.wordpress.com - I made soundcloud-dl.appspot.com - I am a main contributor of youtube-dl. - I teach programming at my school to my friends. - It's my first programming related conference. - The life of a python programmer in Pakistan What this talk is about ? ================== - What is Web Scraping and its usefulness - Which libraries are available for the job - Open Source vs proprietary alternatives - Whaich library is best for which job - When and when not to use Scrapy What is Web Scraping ? ================== Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites. - Wikipedia ###In simple words : It is a method to extract data from a website that does not have an API or we want to extract a LOT of data which we can not do through an API due to rate limiting. We can extract any data through web scraping which we can see while browsing the web. Usage of web scraping in real life. ============================ - to extract product information - to extract job postings and internships - extract offers and discounts from deal-of-the-day websites - Crawl forums and social websites - Extract data to make a search engine - Gathering weather data etc Advantages of Web scraping over using an API ======================== - Web Scraping is not rate limited - Anonymously access the website and gather data - Some websites do not have an API - Some data is not accessible through an API etc Which libraries are available for the job ? ================================ There are numerous libraries available for web scraping in python. Each library has its own weaknesses and plus points. Some of the most widely known libraries used for web scraping are: - BeautifulSoup - html5lib - lxml - re ( not really for web scraping, I will explain later ) - scrapy ( a complete framework ) A comparison between these libraries ============================== - speed - ease of use - what do i prefer - which library is best for which purpose Proprietary alternatives ================== - a list of proprietary scrapers - their price - are they really useful for you ? Working of proprietary alternatives =========================== - how they work (render javascript) - why they are not suitable for you - how custom scrapers beat proprietary alternatives Scrapy ======= - what is it - why is it useful - asynchronous support - an example scraper Question ======= - Questions from the viewers

Martijn Faassen - Morepath: a Python Web Framework with Super Powers [EuroPython 2014] [24 July 2014] Morepath is a server web framework written with modern, rich client web development in mind. Why another new Python web framework in 2014? Because it can be done better: Morepath understands how to construct hyperlinks from models. Writing a generic view in Morepath is like writing any other view. With Morepath, you can reuse, extend and override apps as easily as you can construct them. Even if you don't end up using Morepath, you will learn something about how the nature of web frameworks. ----- [Morepath](http://morepath.readthedocs.org) is a new server web framework written with modern, rich client web development in mind. In the talk I will be discussing some core features of Morepath that make it different: * Its different take on routing and linking. Morepath has support to help you construct hyperlinks to models. * Its view system: plain views, generic views, view composition. * Morepath's approach to application construction allows application extension and overriding, and composition. This talk will attempt to convince people to try Morepath. For those unable or unwilling to try, I will communicate some design principles behind Morepath which can be of help to any web developer.

Niv/tomr - Learning Chess from data [EuroPython 2014] [24 July 2014] Is watching a chess game enough to figure out the rules? What is common denominator between different plays and game ending? In this presentation, we will show how Machine Learning and Hadoop can help us re-discover chess rules and gain new understanding of the game. ----- Can empirical samples unveil the big picture? Is chess games descriptions expose good enough data to gain understanding of chess rules - legal piece moves, castling, check versus checkmate, etc. Which features are important in describing a chess game and which features are not. What is a good representation of a chess game for this uses. What is the minimal sample size which is required in order to learn this in a good enough manner and where this learning can go wrong. **Ne3 => E=mc2** Looking at the bigger picture - Can we understand big systems based on empirical samples. Can we reverse engineer physics and discover how physical system work based on no external knowledge beside empirical samples.

Mauri - VPython goes to School [EuroPython 2014] [22 July 2014] Using VPython in high school is an interesting way to introduce students to get in touch with computer programming concepts and to link computer science with other disciplines like Math, Geometry, Physics, Chemistry ----- My presentation is focused mainly on my teaching experience in a high school using VPython. I've posed some problems to my students to solve with VPython: from basic static building representations like castle to more complex dynamic models like bouncing balls. This approach seems a good way to get in touch with computer programming concepts and to link computer science with other disciplines like Math, Geometry, Physics, Chemistry

Flavio Percoco - Systems Integration: The OpenStack success story [EuroPython 2014] [23 July 2014] OpenStack is a huge, open-source cloud provider. One of the main tenets of OpenStack is the (Shared Nothing Architecture) to which all modules stick very closely. In order to do that, services within OpenStack have adopted different strategies to integrate themselves and share data without sacrificing performance nor moving away from SNA. This strategies are not applicable just to OpenStack but to any distributed system. Sharing data, regardless what that data is, is a must-have requirement of any successful cloud service. This talk will present some of the existing integration strategies that are applicable to cloud infrastructures and enterprise services. The talk will be based on the strategies that have helped OpenStack to be successful and most importantly, scalable. ----- Abstract ======= OpenStack is a huge, open-source cloud provider. One of the main tenets of OpenStack is the (Shared Nothing Architecture) to which all modules stick very closely. In order to do that, services within OpenStack have adopted different strategies to integrate themselves and share data without sacrificing performance nor moving away from SNA. This strategies are not applicable just to OpenStack but to any distributed system. Sharing data, regardless what that data is, is a must-have requirement of any successful cloud service. This talk will present some of the existing integration strategies that are applicable to cloud infrastructures and enterprise services. The talk will be based on the strategies that have helped OpenStack to be successful and most importantly, scalable. Details ====== Along the lines of what I've described in the abstract, the presentation will walk the audience through the state of the art of existing system integration solutions, the ones that have been adopted by OpenStack and the benefits of those solutions. At the end of the talk, a set of solutions under development, ideas and improvements to the existing ones will be presented. The presentation is oriented to distributed services, fault-tolerance and replica determinism. It's based on a software completely written in python and running successfully on several production environments. The presentation will be split in 3 main topics: Distributed System integration ----------------------------------- * What's it ? * Why is it essential for cloud infrastructures? * Existing methods and strategies OpenStack success story ---------------------------- * Which methods did OpenStack adopt? * How / Why do they work? * What else could be done? Coming Next --------------- * Some issues of existing solutions * What are we doing to improve that? * Other solutions coming up

ykaplan - Marconi - OpenStack Queuing and Notification Service [EuroPython 2014] [22 July 2014] Marconi is a multi-tenant cloud queuing system written in Python as part of the OpenStack project. Marconi aims to ease the design of distributed systems and allow for asynchronous work distribution without creating yet another message broker. This talk aims to give the audience a broad look at Marconi’s design and technologies used. ----- Similar to other message bus frameworks, Marconi's main goals are: performance, availability, durability, fault-tolerance and scalability. Besides providing support for queuing and notification services through OpenStack, Marconi aims to ease the design of distributed systems and allow for asynchronous work distribution without creating yet another message broker. This talk aims to give the audience a broad look at Marconi’s architecture, design, technologies used, development process, and discuss the issues it adresses.

Jair Trejo - Non Sequitur: An exploration of Python's random module [EuroPython 2014] [24 July 2014] An exploration of Python's random module for the curious programmer, this talk will give a little background in statistics and pseudorandom number generation, explain the properties of python's choice of pseudorandom generator and explore through visualizations the different distributions provided by the module. ----- # Audience Non mathematical people who wants a better understanding of Python's random module. # Objectives The audience will understand pseudorandom number generators, the properties of Python's Mersenne Twister and the differences and possible use cases between the distributions provided by the `random` module. # The talk I will start by talking about what randomness means and then about how we try to achieve it in computing through pseudorandom number generators (5 min.) I will give a brief overview of pseudorandom number generation techniques, show how their quality can be assessed and finally talk about Python's Mersenne Twister and why it is a fairly good choice. (10 min.) Finally I will talk about how from randomness we can build generators with interesting probability distributions. I'll compare through visualizations thos provided in Python's `random` module and show examples of when they can be useful in real-life. (10 min.)

Katarzyna Jachim - Python in system testing [EuroPython 2014] [23 July 2014] When you think about Python+testing, you usually think about testing your code - unittests, mostly. But it is not the only case! When you have a big system, you need to test it on much higher level - if only to check if all the components are wired in the right way. You may do it manually, but it is tedious and time-consuming - so you want to automate it. And here comes Python - the language of choice in many QA departments. ----- When you think about Python+testing, you usually think about testing your code - unittests, mostly. But it is not the only case! When you have a big system, you need to test it on much higher level - if only to check if all the components are wired in the right way. You may do it manually, but it is tedious and time-consuming - so you want to automate it. And here comes Python - the language of choice in many QA departments. I will tell about differences between unittesting and system testing which result in totally different requirements on test management/running systems. I will tell how we use Python (and a little why) to automate our work. Finally, I will tell a little about my "idee fixe" - a framework for system testing written in Python.

Julian Berman - Design Your Tests [EuroPython 2014] [23 July 2014] While getting started testing often provides noticeable immediate improvement for any developer, it's often not until the realization that tests are things that need design to provide maximal benefit that developers begin to appreciate or even enjoy them. We'll investigate how building shallow, transparent layers for your tests makes for better failures, clearer tests, and quicker diagnoses. ----- * Life span of a test * 5 minute - why does this fail? * 5 day - what is this missing? * 5 week - do I have coverage for this? * 5 month - what's *not* causing this bug? * Transparent simplicity * one or two "iceberg" layers for meaning * Higher-order assertions - build collections of state that have meaning for the domain in the tests * bulk of the details are in the code itself * show an example * grouping for organization * Mixins * show an example * unittest issues * assertion/mixin clutter * setUp/tearDown tie grouping to the class layer or to inheritance via super * addCleanup * weak association / lookup-ability between code and its tests * package layout * other conventions * Alternative approaches * testtools' matchers * py.test `assert` magic

Stefan Behnel - The Cython Compiler for Python [EuroPython 2014] [22 July 2014] The Cython compiler is the most widely used static compiler for Python. It is used to speed up Python code and to extend CPython with fast native extension modules that process huge amounts of data all around the world. This talk by one of the core developers gives an intro to using the compiler and an overview of its major features. ----- The Cython compiler is the most widely used static compiler for Python. The code it generates is used in countless critical applications that process huge amounts of data world wide. Cython has two major use cases: to compile Python code into fast native extension modules, and to connect native code to the CPython runtime. The main goal of the Cython project is to make it easy for users to manually optimise their Python code to make it run at C speed. This talk by one of the core developers will give an intro to using the compiler and an overview of its major features. Outline will be more or less as follows: * Cython: intro to the project and the compiler (4 min.) * compiling Python code - how to do it and what you get (3 min.) - a tiny bit of distutils (2 min.) * static typing and Cython extensions to the Python language - static typing in Cython language syntax (3 min.) - static typing in pure Python syntax (2 min.) - why Cython's type system is cool and what users need to know about it (8 min.) - Cython for optimising Python code (5 min.) * quick intro: talking to native C/C++ code in Cython - using external C APIs (4 min.) - using external C++ APIs (3 min.) - how to build and link in distutils (2 min.) - notes on ways to wrap large C-APIs (1 min.) * quick overview: special features for high-performance code - NumPy integration and memory views, fused types, parallel loops in all brevity (3 min.)

Tomasz Paczkowski - Fun with cPython memory allocator [EuroPython 2014] [25 July 2014] Working with Python does not usually involve debugging memory problems: the interpreter takes care of allocating and releasing system memory and you get to enjoy working on real world issues. But what if you encounter such problems? What if your program never releases memory? How do you debug it? This talk describes some of the lesser known properties of cPython memory allocator and some ways to debug memory-related problems, all this based on real events. ----- Working with Python does not usually involve debugging memory problems: the interpreter takes care of allocating and releasing system memory and you get to enjoy working on real problems. But what if you encounter such problems? What if your program never releases memory? How do you debug it? I will tell a story of one programmer discovering such problems. The talk will take listeners on a journey of issues they can encounter, tools they can use to debug the problems and possible solutions to seek out. There will also be a brief mention of general memory management principles. cPython uses a combination of its own allocator, `malloc`, and `mmap` pools to manage memory of Python programs. It usually is smart enough, but there are some darker corners that are not well known by an average Joe Programmer (read: me). There are tools that can help debug memory problems, but those are also relatively unknown, and tend to have documentation that one might find lacking. I will describe one such tool, called `guppy`, which I have found particulary helpful.

Andreas Klostermann - Brain Waves for Hackers [EuroPython 2014] [22 July 2014] Using the Neurosky Mindwave headset we can read brain waves via bluetooth. In this talk I will present ways to interface with this device and to analyze their data, mostly from the perspective of a Python hacker, and not so much from the perspective of academic research or the development of production-ready software. I will convey the basic scientific and technical background, keeping in mind the non-medical audience, and touch on several practical applications, especially Neurofeedback and its use for personal mental training and potential therapeutic uses. The Audience can expect to walk away with a basic knowledge of EEG analysis, Neurofeedback and how to start experimenting with these devices on their own. ----- A new class of cheap consumer EEG devices allows ordinary hackers and even high school students a glimpse into the human brain. This talk will present how to use the "Neurosky Mindwave" headset with python software, and lay out the basic scientific and technical background. The Mindwave Mobile is a device that can be easily talked to using bluetooth, and it talks a binary protocol which is specifically designed to be useful without much computing power in the receiving device or advanced knowledge of signal processing. In fact, an Arduino with a few lines of code is perfectly capable of parsing some of the byte stream and reacting to the mental state of the user, while fully-featured python software can do advanced analysis using PyEEG and Pandas. The same hardware module and protocol is used in the Nekomimi headset (mind-controlled cat ears for cosplay) and some Boardgames (MindFlex). A python library for interfacing with the headset is presented and will be demonstrated on stage. Mostly kivy applications will be used. Also I will present some data analysis you can perform with pandas and scipy. Neurofeedback is a type of mental exercise where a computer uses the EEG data to direct the user towards certain mental states. In the easiest configuration a program would display a bar with the "concentration" level, and the user would learn how to tilt this bar upwards. In more complicated setups a game could react favorably towards states like relaxation or concentration. Using Gamification, Neurofeedback can provide a more engaging experience for children or adults, than other techniques with similar goals, like mindfulness meditation, and the more immediate feedback should enhance the effectiveness of mental training, though that has not been investigated scientifically yet. Neurofeedback has been shown to be effective (albeit not recommended as sole treatment) in Patients with Attention Deficit Hyperactivity Disorder (ADHD), Epilepsy and Dementia. Some background about these conditions and applications of Neurofeedback to them will be given. The first use of Neurofeedback was done in Cats, during early experiments with EEG electrodes in the 60ies. Cats where conditioned to exhibit certain wave patterns, and later, due to a coincidence, the researchers noticed that the conditioned cats where more resistant to epilepsy-inducing medications. The effect has since been reproduced in humans, in cases where medications did not work sufficiently. Ample hints on not to treat any of this information as medical advice will be provided. The goal of this talk is to promote Neurofeedback as a useful mental training and to encourage development of applications around Neurofeedback and the analysis of EEG data, from the perspective of a python hacker. I gave a similar talk at PyConDe 2013 in Cologne. The new talk will be in English, show some improvements on the software, and more advanced demonstrations.

Alexey Malashkevich - How Pony ORM translates Python generators to SQL queries [EuroPython 2014] [24 July 2014] Pony ORM is an Object-Relational Mapper implemented in Python. It uses an unusual approach for writing database queries using Python generators. Pony analyzes the abstract syntax tree of a generator and translates it to its SQL equivalent. The translation process consists of several non-trivial stages. In this talk one of Pony ORM authors will reveal the internal details of this process. ----- [Pony ORM](http://ponyorm.com) is an object-relational mapper implemented in Python. It allows writing advanced queries to a database using plain Python in the form of a generator expression. This way queries look very concise. The main feature of Pony is to provide a method to write declarative queries to databases in pure Python using generators. For this purpose Pony analyzes the abstract syntax tree of a generator and translates it to its SQL equivalent. Following is a sample of a query in Pony: select(p for p in Product if "iPad" in p.name and p.price >= 500) This query translates to SQL using a specific database dialect. Currently Pony works with SQLite, MySQL, PostgreSQL and Oracle databases. In this talk one of Pony ORM authors will go through the process of the query translation and dig into the implementation details. Attendees are going to walk away with the understanding of: 1. Principles of building a programming language translator 2. Python to SQL translator implementation details 3. Approaches for creating a pluggable translator architecture The presentation outline: - Why Python generators are good for representing SQL queries - Main stages of Python to SQL translation overview - Decompiling Python bytecode into Python AST - Translating Python AST to database-independent SQL representation - Generating SQL for specific database - Pluggable translator architecture - Performance concerns: is such way of building SQL slower or faster then Django's and SQLAlchemy's?

Lynn Root - For lack of a better name(server): DNS Explained [EuroPython 2014] [23 July 2014] You've deployed! But your friends can't see it - what's wrong? I'm betting DNS. Maybe you've fixed a couple of entries, point some records to hostnames, waited patiently for new domains to resolve only to notice your nameservers are incorrect. But what actually goes on with DNS? Come to this talk to find out how DNS works, and how to interact and create a DNS with Python. ----- Following instructions of what entries to create where is easy enough when using a PaaS. But DNS is hard – deployment issues always seem to come down to DNS. A solid understanding of DNS will not only help with deploying your applications, but will also give a greater understanding of how the internet works, and more generally, distributed systems. In this talk, you will learn what DNS is, how it works and how to communicate with it, and how Python can make both interacting and spinning up your own DNS server simple (I swear!). Outline: * Intro (1-2m) * What DNS is (5 min) * URL -> IP addr, e.g. "phonebook" lookup (obligatory pun: Call me, Maybe?) * hierarchical system & resolution sequence (local DNS cache/resolver, ISP resolver, recursive DNS search) * popular types (primary, secondary/slave, forwarding, authoritative only, etc) * System components: what makes a DNS? * How to communicate with DNS (3 min) * Protocol: UDP * Operations: CRUD * Resource records (A, AAAA, CNAME, SOA, SRV, etc) * tools: dig/nsupdate/nslookup * Security overview (3min) (disclaimer: NOT a DNS security expert, not planning to get into the details here) * Server-Server, DynDNS: TSIG/GSS-TSIG * Server-Client: DNSSEC * Python + DNS (10 min) * plain UDP query in Python (no 3rd-party libraries/no magic) * Interacting with a DNS w/ Python (dnspython.py) * Sample DNS server with Twisted * "fake" demo (either local or pre-recorded screen cast) of querying/updating/etc of the Twisted DNS * Wrap up - resources page, github links, etc (1min) * Q&A - ~5 min

Hynek - The Sorry State of SSL [EuroPython 2014] [22 July 2014] Those web pages with shiny lock icons boasting that your data is safe because of “256 bit encryption”? They are lying. In times of mass surveillance and commercialized Internet crime you should know why that’s the case. This talk will give you an overview that will help you to assess your personal security more realistically and to make your applications as secure as possible against all odds. ----- The rule of thumb for people without degrees in cryptography on securing data on the Internet is “GPG for data at rest. TLS for data in motion”. And it’s actually a very good rule everyone should follow. The only kicker though is that configuring (and using!) TLS properly is not as simple as it sounds and if you’re not diligent as a user, developer, and ops engineer, you can easily compromise your data’s security despite best effort of everyone else. This talk will be multifaceted; you will learn: - how SSL and TLS roughly work and why their state is sorry, - server- and client-side duties for best possible security, - what alternatives you have for using TLS in Python, - things to keep in mind when configuring servers, - and what perils outside your control still can trip you up. In other words, the leitmotif is to show you the most common traps you should know about when using and deploying applications relying on TLS for transport layer security and how to avoid them.

Honza Král - Lessons learned from building Elasticsearch client [EuroPython 2014] [24 July 2014] Lessons learned when building a client for a fully distributed system and trying to minimize context-switching pains when using multiple languages. ----- Last year we decided to create official clients for the most popular languages, Python included. Some of the goals were: * support the complete API of elasticsearch including all parameters * provide a 1-to-1 mapping to the rest API to avoid having opinions and provide a familiar interface to our users consistent across languages and evironments * degrade gracefully when the es cluster is changing (nodes dropping out or being added) * flexibility - allow users to customize and extend the clients easily to suit their, potentially unique, environment In this talk I would like to take you through the process of designing said client, the challenges we faced and the solutions we picked. Amongst other things I will touch on the difference between languages (and their respective communities), the architecture of the client itself, mapping out the API and making sure it stays up to date and integrating with existing tools.

plaetzchen - Don't fear our new robot overlords! [EuroPython 2014] [23 July 2014] This talk will show you GoldenEye. A setup of a robot, image recognition and Python. What could be better? GoldenEye uses computer vision to detect icons on an attached mobile device and then tells a robot to tap them. All off these tests are written in python's unittest module and the API is as easy as possible to make writing complex tests an ease. ----- GoldenEye is our solution for mobile front end tests. Testing on mobile devices can be quite devastating: On iOS you can write front test in JavaScript in Instruments but it is quite impossible to connect Instruments to you CI solution of choice. On Android the situation isn't much better. Other front end test frameworks can work with mobile devices (or simulators) but they lack the ability to see. Of course you can check if a color is set correctly, if a frame has the right x and y coordinates but in a world of different screen sizes writing these tests can be quite challenging as well. In the end you will always need to look on your screen again and again trying to spot any issues. GoldenEye takes a different approach. It does not need to run on your development computer, you don't need a Mac for running tests on iOS devices and you can have real touches on your controls. This is archived by using openCV and it's python bindings, Pythons's unittest module and the Tapsterbot, an OpenSource delta robot made with 3D printing and an Arduino controller. To write a test you just take some screenshots on your device, cut out the icons you need to tap or inspect and write a very simple unit test using a high-level API that takes away the hard parts. WARNING: This talk features a real robot. In case of machine world-domination: RUN!

Conference Closing [EuroPython 2014] [25 July 2014]