AG Euler Wiki - User contributions [en]

Email Addresses

2018-01-18T14:41:05Z

Lrogerson:

Thomas Euler <thomas.euler@cin.uni-tuebingen.de>

Valeska Botzenhardt <valeska.botzenhardt@cin.uni-tuebingen.de>

Timm Schubert <timm.schubert@googlemail.com>

Tom Baden <thomas.baden@uni-tuebingen.de>

Zhijian Zhao <zzj1031@gmail.com>

Philipp Berens <philipp.berens@uni-tuebingen.de>

Gordon Eske <gordon.eske@klinikum.uni-tuebingen.de>

Katrin Franke <katrin.franke@cin.uni-tuebingen.de>

Miroslav Román Rosón <miroslav.roman.roson@gmail.com>

Luke Rogerson <luke.e.rogerson@gmail.com>

Yanli Ran <ran.yyyy@163.com>

Michael Power <michaeljtpower@gmail.com>

Yannik Bauer <bauer.yannik@googlemail.com>

Email Addresses

2018-01-18T14:40:52Z

Lrogerson:

Thomas Euler <thomas.euler@cin.uni-tuebingen.de>

Valeska Botzenhardt <valeska.botzenhardt@cin.uni-tuebingen.de>

Timm Schubert <timm.schubert@googlemail.com>

Tom Baden <thomas.baden@uni-tuebingen.de>

Zhijian <zzj1031@gmail.com>

Philipp Berens <philipp.berens@uni-tuebingen.de>

Gordon Eske <gordon.eske@klinikum.uni-tuebingen.de>

Katrin Franke <katrin.franke@cin.uni-tuebingen.de>

Miroslav Román Rosón <miroslav.roman.roson@gmail.com>

Manoj Kulkarni <manoj.kulkarni@uni-tuebingen.de>

Camille Chapot <camille_chapot@hotmail.com>

Luke Rogerson <luke.e.rogerson@gmail.com>

Yanli Ran <ran.yyyy@163.com>

Michael Power <michaeljtpower@gmail.com>

Yannik Bauer <bauer.yannik@googlemail.com>

MATLAB

2018-01-18T12:57:11Z

Lrogerson:

I heard that a couple of people in the lab were hoping to use MATLAB for various things. Following a discussion with IT I've been shown how to access MATLAB from any computer with internet access, including workstations in the CIN and computers outside the local network.

I've written short instructions below on how to do this for a CIN computer, but if you want to run this from home drop me a message. It is not necessary to install MATLAB on your computer for this to work, though you will need to install Putty and Xming. Setup time for this should be around 10 to 20 minutes.

Note that programs executed on MATLAB this way should be fast, as they are running on the CIN's high performance computing cluster. You may experience a slight delay with MATLAB responding to your typing, which will depend on the speed of your connection (fast in the CIN, slower remotely). It is possible to set Xming and Putty up on a memory stick to make this method portable - let me know if you want to do this.

'''Installation Instructions'''

Putty: chiark.greenend.org.uk/~sgtatham/putty/download.html
Xming: straightrunning.com/xmingnotes/

Install putty. Install Xming. Run Xming. Ignore the various options (set them as you want) and click through until it minimises to your toolbar.

Run Putty. Set the host name to 172.25.250.112, port 60222. Save the session with a suitable name. Click the + next to ssh on the left hand toolbar, click X11, tick 'Enable X11 forwarding', set the X display location to 'localhost:0'. Return to the sessions page and save the session.

Select your session and click Open. This will open a terminal window, which will request your CIN username (initial and surname, e.g. 'lrogerson') and password. Enter these and it should return a message saying you are at the CIN Cluster headnode with a picture of a startled mouse.

Enter either 'ssh cn30 -X' or 'ssh cn31 -X' (would prefer if you used cn30, but it doesn't matter very much). This will take you to the Euler lab nodes, which are environments that people in the lab can use for running programs on the supercomputers.

Here, enter '/usr/local/MATLAB/R2015a/bin/matlab', which will run an instance of the latest edition of MATLAB. If X11 & Xming are set up correctly, after a few seconds a MATLAB window will open on your computer, which works like a typical local installation. This reads files and scripts from your personal folder on the CIN storage.

MATLAB

2018-01-18T12:56:52Z

Lrogerson:

I heard that a couple of people in the lab were hoping to use MATLAB for various things. Following a discussion with IT I've been shown how to access MATLAB from any computer with internet access, including workstations in the CIN and computers outside the local network.

I've written short instructions below on how to do this for a CIN computer, but if you want to run this from home drop me a message. It is not necessary to install MATLAB on your computer for this to work, though you will need to install Putty and Xming. Setup time for this should be around 10 to 20 minutes.

Note that programs executed on MATLAB this way should be fast, as they are running on the CIN's high performance computer. You may experience a slight delay with MATLAB responding to your typing, which will depend on the speed of your connection (fast in the CIN, slower remotely). It is possible to set Xming and Putty up on a memory stick to make this method portable - let me know if you want to do this.

'''Installation Instructions'''

Putty: chiark.greenend.org.uk/~sgtatham/putty/download.html
Xming: straightrunning.com/xmingnotes/

Install putty. Install Xming. Run Xming. Ignore the various options (set them as you want) and click through until it minimises to your toolbar.

Run Putty. Set the host name to 172.25.250.112, port 60222. Save the session with a suitable name. Click the + next to ssh on the left hand toolbar, click X11, tick 'Enable X11 forwarding', set the X display location to 'localhost:0'. Return to the sessions page and save the session.

Select your session and click Open. This will open a terminal window, which will request your CIN username (initial and surname, e.g. 'lrogerson') and password. Enter these and it should return a message saying you are at the CIN Cluster headnode with a picture of a startled mouse.

Enter either 'ssh cn30 -X' or 'ssh cn31 -X' (would prefer if you used cn30, but it doesn't matter very much). This will take you to the Euler lab nodes, which are environments that people in the lab can use for running programs on the supercomputers.

Here, enter '/usr/local/MATLAB/R2015a/bin/matlab', which will run an instance of the latest edition of MATLAB. If X11 & Xming are set up correctly, after a few seconds a MATLAB window will open on your computer, which works like a typical local installation. This reads files and scripts from your personal folder on the CIN storage.

Main Page

2015-12-04T18:18:48Z

Lrogerson: /* Computational */

Welcome to the AG Euler wiki.
Feel free to contribute additional content.

==General==

[[Email Addresses]]

[[Internal Phone Numbers]]

==Experimental==

[[Ringer Solution]]

==Computational==

{|
|[[Cluster]] ..||[[Datajoint]] ..||[[Docker]] ..||[[Jupyter]] ..||[[MATLAB]] ..||[[Python]]
|}

MATLAB

2015-12-04T18:17:44Z

Lrogerson: Created page with "I heard that a couple of people in the lab were hoping to use MATLAB for various things. Following a discussion with IT I've been shown how to access MATLAB from any computer ..."

I heard that a couple of people in the lab were hoping to use MATLAB for various things. Following a discussion with IT I've been shown how to access MATLAB from any computer with internet access, including workstations in the CIN and computers outside the local network.

I've written short instructions below on how to do this for a CIN computer, but if you want to run this from home drop me a message. It is not necessary to install MATLAB on your computer for this to work, though you will need to install Putty and Xming. Setup time for this should be around 10 to 20 minutes.

Note that programs executed on MATLAB this way should be very, very fast, as they are running on the CIN's high performance computer. You may experience a slight delay with MATLAB responding to your typing, which will depend on the speed of your connection (fast in the CIN, slower remotely). It is possible to set Xming and Putty up on a memory stick to make this method portable - let me know if you want to do this.

'''Installation Instructions'''

Putty: chiark.greenend.org.uk/~sgtatham/putty/download.html
Xming: straightrunning.com/xmingnotes/

Install putty. Install Xming. Run Xming. Ignore the various options (set them as you want) and click through until it minimises to your toolbar.

Run Putty. Set the host name to 172.25.250.112, port 60222. Save the session with a suitable name. Click the + next to ssh on the left hand toolbar, click X11, tick 'Enable X11 forwarding', set the X display location to 'localhost:0'. Return to the sessions page and save the session.

Select your session and click Open. This will open a terminal window, which will request your CIN username (initial and surname, e.g. 'lrogerson') and password. Enter these and it should return a message saying you are at the CIN Cluster headnode with a picture of a startled mouse.

Enter either 'ssh cn30 -X' or 'ssh cn31 -X' (would prefer if you used cn30, but it doesn't matter very much). This will take you to the Euler lab nodes, which are environments that people in the lab can use for running programs on the supercomputers.

Here, enter '/usr/local/MATLAB/R2015a/bin/matlab', which will run an instance of the latest edition of MATLAB. If X11 & Xming are set up correctly, after a few seconds a MATLAB window will open on your computer, which works like a typical local installation. This reads files and scripts from your personal folder on the CIN storage.

Email Addresses

2015-12-04T18:16:32Z

Lrogerson:

Thomas Euler <thomas.euler@cin.uni-tuebingen.de>

Valeska Botzenhardt <valeska.botzenhardt@cin.uni-tuebingen.de>

Timm Schubert <timm.schubert@googlemail.com>

Tom Baden <thomas.baden@uni-tuebingen.de>

Zhijian <zzj1031@gmail.com>

Philipp Berens <philipp.berens@uni-tuebingen.de>

Gordon Eske <gordon.eske@klinikum.uni-tuebingen.de>

Katrin Franke <katrin.franke@cin.uni-tuebingen.de>

Miroslav Román Rosón <miroslav.roman.roson@gmail.com>

Manoj Kulkarni <manoj.kulkarni@uni-tuebingen.de>

Camille Chapot <camille_chapot@hotmail.com>

Luke Rogerson <luke.e.rogerson@gmail.com>

Yanli Ran <ran.yyyy@163.com>

Michael Power <michaeljtpower@gmail.com>

Theresa Stadler <ths.stadler@googlemail.com>

'''Alumni'''

Yannik Bauer <bauer.yannik@googlemail.com>

Joe Sheppard <joe-sheppard@outlook.com>

Main Page

2015-10-13T16:07:17Z

Lrogerson: /* Computational */

Main Page

2015-10-13T16:06:56Z

Lrogerson: /* Computational */

Welcome to the AG Euler wiki.
Feel free to contribute additional content.

==General==

[[Email Addresses]]

[[Internal Phone Numbers]]

==Experimental==

[[Ringer Solution]]

==Computational==

{|
|[[Cluster]]||[[Datajoint]]||[[Docker]]||[[Jupyter]]||[[Python]]
|}

Python

2015-10-13T15:44:52Z

Lrogerson: /* Resources For Learning Python */

===Resources For Learning Python===

Though not comparable in size to the truly enormous userbase of C++, Java and C#, Python still boasts a highly active community and a correspondingly large number of resources. In the Euler group all users are strongly encouraged to use Python 3. While some modules only exist for the older Python 2 codebase, in the overwhelming majority of cases they have been re-established in the Py3 framework (cf. [https://python3wos.appspot.com/ Python 3 Wall Of Superpowers] for current status of module transfer).

For the absolute beginner Codeacademy's free, interactive [https://www.codecademy.com/en/tracks/python learning module for Python] is a good starting point. Though it recommends approximately 13 hours for the course, those with a background in programming could probably finish it in under a day. For scientific computing it helps to have some familiarity with the Scipy stack, for which there is a nice introduction at [http://www.scipy-lectures.org/ scipy-lectures.org], particularly the Getting Started section.

Beyond that, it will be worth taking a look at the [https://web.stanford.edu/~mwaskom/software/seaborn/ Seaborn] module for attractive plotting functions. Those interested in Datajoint should also take a look at the [https://datajoint.github.io/ project page] and that of [https://pythonhosted.org/neo/ Neo], the latter of which may be used for Python IO. Finally, if you are thinking of using Python for important code which will likely be reused, I would recommend reading the [https://www.python.org/dev/peps/pep-0008/ PEP0008 styleguide], to ensure you are writing correctly formatted code.

Jupyter

2015-10-13T15:43:52Z

Lrogerson: /* Which notebook servers are currently running? */

==What are Jupyter Notebooks?==

Jupyter notebooks are browser based environments for editing, compiling and executing code. For a brief overview, see [http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 this Nature News article]. Until quite recently, these were developed exclusively for coding Python, although support has since expanded to accommodate a number of other languages. For a full list of language kernels, see [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages here]. Note that Igor Pro is not supported, and that many of these kernels are in alpha state and not suitable for daily use.

==Which notebook servers are currently running?==
We are currently running two Jupyter notebook servers in the CIN network, dubbed [https://en.wikipedia.org/wiki/Callisto_(moon) Callisto] and [https://en.wikipedia.org/wiki/Titan_(moon) Titan]. Callisto is geared towards stability and is the recommended environment for those who wish to write standard Python scripts. It comes with the [https://www.scipy.org/about.html scipy stack] preinstalled, which should be adequate for most scientific computing problems. The Callisto server will only be taken down after 6pm on Fridays, if at all, and should be functional again by the following morning.

Alongside this is a more experimental notebook server called Titan, which will be used for testing advanced features which accelerate the performance of code in Python (LLVM/Numba, Spark, CUDA, etc). Titan will be down more frequently as new features are implemented and tested. If you have any features you would like to try out let me know (luke.e.rogerson@gmail.com). If the implementations are sufficiently stable, we can migrate the content to Callisto at a later date.

==How do I access the AG Euler Jupyter Notebooks?==

To access the notebooks, you will have to specify the correct Putty configuration. A preconfigured file is available, and details for manual set up are given below.
If you forget the access details - each server is named after a moon, the year of its discovery is the localhost number, and the password is the planet which the moon orbits. If at any time you cannot remember the password or localhost address, refer to the wikipedia pages for the namesakes of the respective servers.

===Preconfigured Portable Putty===
[[File:AG Euler - Portable Putty - 13.10.2015.zip]]

===Manually Configure Putty===

#Open Putty, load your Cluster access configuration
#Under Category, click the '+' to expand ssh
#Click tunnels, and add the following:
##Destination 172.29.0.131:1610 .. Source: 1610
##Destination 172.29.0.131:1655 .. Source: 1655
#Save your configuration in the session tab
#Open the updated configuration by double clicking on it
#Use your CIN user account to log into the headnode
#Enter "ssh 'cn31'" to access the Euler group node where the servers are running
#Enter either https://localhost:1610 or https://localhost:1655 to access Callisto or Titan respectively. This should then take you to a login screen (or directly to the directories).
# Enter the password for the server (Case sensitive - Callisto: jupiter, Titan: saturn)

Jupyter

2015-10-13T15:43:37Z

Lrogerson: /* Which notebook servers are currently running? */

==What are Jupyter Notebooks?==

Jupyter notebooks are browser based environments for editing, compiling and executing code. For a brief overview, see [http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 this Nature News article]. Until quite recently, these were developed exclusively for coding Python, although support has since expanded to accommodate a number of other languages. For a full list of language kernels, see [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages here]. Note that Igor Pro is not supported, and that many of these kernels are in alpha state and not suitable for daily use.

==Which notebook servers are currently running?==
We are currently running two Jupyter notebook servers in the CIN network, dubbed [https://en.wikipedia.org/wiki/Callisto_(moon) Callisto] and [https://en.wikipedia.org/wiki/Titan_(moon) Titan]. Callisto is geared towards stability and is the recommended environment for those who wish to write standard Python scripts. It comes with the [https://www.scipy.org/about.html scipy stack] preinstalled, which should be adequate for most scientific computing problems. The Callisto server will only be taken down after 6pm on Fridays, if at all, and should be functional again by the following morning.

Alongside this is a more experimental notebook server called Titan, which will be used for testing advanced features which accelerate the performance of code in Python (LLVM/Numba, Spark, CUDA, etc). Titan will be down more frequently as new features are implemented and tested. If you have any features you would like to try out, or have implemented, let me know (luke.e.rogerson@gmail.com). If the implementations are sufficiently stable, we can migrate the content to Callisto at a later date.

==How do I access the AG Euler Jupyter Notebooks?==

To access the notebooks, you will have to specify the correct Putty configuration. A preconfigured file is available, and details for manual set up are given below.
If you forget the access details - each server is named after a moon, the year of its discovery is the localhost number, and the password is the planet which the moon orbits. If at any time you cannot remember the password or localhost address, refer to the wikipedia pages for the namesakes of the respective servers.

===Preconfigured Portable Putty===
[[File:AG Euler - Portable Putty - 13.10.2015.zip]]

===Manually Configure Putty===

#Open Putty, load your Cluster access configuration
#Under Category, click the '+' to expand ssh
#Click tunnels, and add the following:
##Destination 172.29.0.131:1610 .. Source: 1610
##Destination 172.29.0.131:1655 .. Source: 1655
#Save your configuration in the session tab
#Open the updated configuration by double clicking on it
#Use your CIN user account to log into the headnode
#Enter "ssh 'cn31'" to access the Euler group node where the servers are running
#Enter either https://localhost:1610 or https://localhost:1655 to access Callisto or Titan respectively. This should then take you to a login screen (or directly to the directories).
# Enter the password for the server (Case sensitive - Callisto: jupiter, Titan: saturn)

Cluster

2015-10-13T15:41:47Z

Lrogerson: /* Preconfigured Portable Putty */

==What is the cluster?==

The CIN cluster is a High Performance Computing environment curated by the CIN IT services. It is composed of many 'nodes', which are computational environments which you can access both from your workstation and remotely, each of which can be used to run tasks independently. For computationally intensive work, it is recommended that you run your programs on this cluster. For the curious, the cluster is run on the [http://www.ch.cam.ac.uk/computing/maui-and-torque-introduction Maui/Torque architecture].

==How do I access the cluster?==

The cluster is accessed using a program called Putty. The [https://cinwiki/mediawiki/cinwiki/index.php/Storage_cluster instructions on the CIN wiki] should be sufficient for gaining access. In order to use the cluster, you will require a CIN user account, the name of which looks something like ''kelch'' for the user Karl Elch. It is possible to access the cluster remotely, and so execute code in the system from outside the CIN. To do so, follow the [https://cinwiki/mediawiki/cinwiki/index.php/VPN instructions on the CIN wiki]. Our nodes are 'cn30' and 'cn31', the latter of which is used for our Jupiter server. It is also possible to access GPU resources at 'gpu1', 'gpu2' and 'gpu3', but keep in mind that these are for general use by all groups in the CIN and may be engaged.

===Preconfigured Portable Putty===

Alternatively, you can access a preconfigured version of Putty intended for use with the Jupyter servers.

[[File:AG Euler - Portable Putty - 13.10.2015.zip]]

Docker (deprecated)

2015-10-13T15:19:16Z

Lrogerson: /* How do I build my own docker containers? */

==What is docker?==

Docker is a platform for running virtual machines called ''containers''. These are functionally analogous to the operating systems on a computer, though you can run them from within your operating system. They are useful because they create lightweight, customisable environments within which you can run software. This is particularly helpful when you have programs which have very particular dependencies, or require a linux base installation in order to function correctly. You can read more about Docker [https://www.docker.com/whatisdocker here].

==What do we use docker for?==

The Euler group is currently using docker to host [[Jupyter_Notebooks|our two Jupyter notebook servers]], though in the past we have also used it for hosting a MySQL server. To see what containers are currently running, ssh into 'cn31' and enter the command: <blockquote>docker ps</blockquote> This will return a list of the currently active docker containers, the images they are built from, how long they have been online and what ports they are forwarded to.

==How do I build my own docker containers?==
New docker containers can be built using the command: <blockquote>docker run [-tags] [imageName]</blockquote> For example, to build a new scipyserver called 'miranda', forwarded to port 1948 on localhost with the password 'uranus', you would enter: <blockquote>docker run -d -p 1948:8888 --name miranda -e "PASSWORD=uranus" -e DISPLAY=$DISPLAY ipython/scipyserver</blockquote> Each of the -tags has a unique function in building the server. The last entry specifies the image from which the container is built, in this case an ipython server. You can find more images over at [https://hub.docker.com/ Dockerhub].

<blockquote>
{|class="wikitable"
|'''Tag'''||'''Function'''
|-
| -d||Run in the background
|-
| -p 1948:8888||Forward port 8888 to 1948
|-
| --name miranda||Specify container name as miranda
|-
| -e "PASSWORD=uranus"||Set password to uranus
|-
| -e DISPLAY=$DISPLAY||Needed for plotting in Jupyter
|-
| -v||Mount volume, see below
|-
| --help||Show different tag options for given command
|} </blockquote>

Data stored in docker containers is transient and is lost when the container is deleted. In order to set the containers so that they can access volumes on the host system and persist their content after they are deleted, we have to use additional tags to mount directories in the container to directories on the host system. Here is an example of this for another ipython server:<blockquote>docker run -v ~/Notebooks:/notebooks/Notebooks -v ~/Data:/notebooks/Data:ro -v /tmp/.X11-unix:/tmp/.X11-unix:ro ipython/scipyserver</blockquote> Here, three different directories have been mounted: a volume which the container can write to, under notebooks/Notebooks, a volume which the container can read, under notebooks/Data (the :ro tag specifies read only), and a further folder used for X11 forwarding, which is necessary for getting figures in the Jupiter server.

For a more thorough walkthough, see the official [https://docs.docker.com/linux/started/ Getting Started] documentation.

Docker (deprecated)

2015-10-13T15:18:54Z

Lrogerson: /* How do I build my own docker containers? */

==What is docker?==

Docker is a platform for running virtual machines called ''containers''. These are functionally analogous to the operating systems on a computer, though you can run them from within your operating system. They are useful because they create lightweight, customisable environments within which you can run software. This is particularly helpful when you have programs which have very particular dependencies, or require a linux base installation in order to function correctly. You can read more about Docker [https://www.docker.com/whatisdocker here].

==What do we use docker for?==

The Euler group is currently using docker to host [[Jupyter_Notebooks|our two Jupyter notebook servers]], though in the past we have also used it for hosting a MySQL server. To see what containers are currently running, ssh into 'cn31' and enter the command: <blockquote>docker ps</blockquote> This will return a list of the currently active docker containers, the images they are built from, how long they have been online and what ports they are forwarded to.

==How do I build my own docker containers?==
New docker containers can be built using the command: <blockquote>docker run [-tags] [imageName]</blockquote> For example, to build a new scipyserver called 'miranda', forwarded to port 1948 on localhost with the password 'uranus', you would enter: <blockquote>docker run -d -p 1948:8888 --name miranda -e "PASSWORD=uranus" -e DISPLAY=$DISPLAY ipython/scipyserver</blockquote> Each of the -tags has a unique function in building the server. The last entry specifies the image from which the container is built, in this case an ipython server. You can find more images over at [https://hub.docker.com/ Dockerhub].

<blockquote>
{|class="wikitable"
|'''Tag'''||'''Function'''
|-
| -d||Run in the background
|-
| -p 1948:8888||Forward port 8888 to 1948
|-
| --name miranda||Specify container name as miranda
|-
| -e "PASSWORD=uranus"||Set password to uranus
|-
| -e DISPLAY=$DISPLAY||Needed for plotting in Jupyter
|-
| --help||Show different tag options for given command
|-
| -v||Mount volume, see below
|} </blockquote>

Data stored in docker containers is transient and is lost when the container is deleted. In order to set the containers so that they can access volumes on the host system and persist their content after they are deleted, we have to use additional tags to mount directories in the container to directories on the host system. Here is an example of this for another ipython server:<blockquote>docker run -v ~/Notebooks:/notebooks/Notebooks -v ~/Data:/notebooks/Data:ro -v /tmp/.X11-unix:/tmp/.X11-unix:ro ipython/scipyserver</blockquote> Here, three different directories have been mounted: a volume which the container can write to, under notebooks/Notebooks, a volume which the container can read, under notebooks/Data (the :ro tag specifies read only), and a further folder used for X11 forwarding, which is necessary for getting figures in the Jupiter server.

For a more thorough walkthough, see the official [https://docs.docker.com/linux/started/ Getting Started] documentation.

File:CIN Bedarfsmeldung Verbrauchsgüter.pdf

2015-10-13T15:03:54Z

Lrogerson: Form for ordering books for the lab. Coordinate with Gordon and ensure you have permission before making an order.

Form for ordering books for the lab. Coordinate with Gordon and ensure you have permission before making an order.

Main Page

2015-10-13T15:02:26Z

Lrogerson: /* Computational */

Welcome to the AG Euler wiki.
Feel free to contribute additional content.

==General==

[[Email Addresses]]

[[Internal Phone Numbers]]

==Experimental==

[[Ringer Solution]]

==Computational==

[[Cluster]]

[[Datajoint]]

[[Docker]]

[[Jupyter]]

[[Python]]

Python

2015-10-13T15:02:12Z

Lrogerson: /* Resources For Learning Python */

===Resources For Learning Python===

Though not comparable in size to the truly enormous userbase of C++, Java and C#, Python still boasts a highly active community and a correspondingly large number of resources. In the Euler group all users are strongly encouraged to use Python 3. While some modules only exist for the older Python 2 codebase, in the overwhelming majority of cases they have been re-established in the Py3 framework (cf. [https://python3wos.appspot.com/ Python 3 Wall Of Superpowers] for current status of module transfer).

For the absolute beginner Codeacademy's free, interactive [https://www.codecademy.com/en/tracks/python learning module for Python] is a good starting point. Though it recommends approximately 13 hours for the course, those with a background in programming could probably finish it in under a day. For scientific computing it helps to have some familiarity with the Scipy stack, for which there is a nice introduction at [http://www.scipy-lectures.org/ scipy-lectures.org], particularly the Getting Started section.

Beyond that, it will be worth taking a look at the [https://web.stanford.edu/~mwaskom/software/seaborn/ Seaborn] module for attractive plotting functions. Those interested in Datajoint should also take a look at the Datajoint project page and that of [https://pythonhosted.org/neo/ Neo], the latter of which may be used for Python IO. Finally, if you are thinking of using Python for important code which will likely be reused, I would recommend reading the [https://www.python.org/dev/peps/pep-0008/ PEP0008 styleguide], to ensure you are writing correctly formatted code.

Python

2015-10-13T15:02:01Z

Lrogerson: /* Resources For Learning Python */

===Resources For Learning Python===

Though not comparable in size to the truly enormous userbase of C++, Java and C#, Python still boasts a highly active community and a correspondingly large number of resources. In the Euler group all users are encouraged to use Python 3. While some modules only exist for the older Python 2 codebase, in the overwhelming majority of cases they have been re-established in the Py3 framework (cf. [https://python3wos.appspot.com/ Python 3 Wall Of Superpowers] for current status of module transfer).

For the absolute beginner Codeacademy's free, interactive [https://www.codecademy.com/en/tracks/python learning module for Python] is a good starting point. Though it recommends approximately 13 hours for the course, those with a background in programming could probably finish it in under a day. For scientific computing it helps to have some familiarity with the Scipy stack, for which there is a nice introduction at [http://www.scipy-lectures.org/ scipy-lectures.org], particularly the Getting Started section.

Beyond that, it will be worth taking a look at the [https://web.stanford.edu/~mwaskom/software/seaborn/ Seaborn] module for attractive plotting functions. Those interested in Datajoint should also take a look at the Datajoint project page and that of [https://pythonhosted.org/neo/ Neo], the latter of which may be used for Python IO. Finally, if you are thinking of using Python for important code which will likely be reused, I would recommend reading the [https://www.python.org/dev/peps/pep-0008/ PEP0008 styleguide], to ensure you are writing correctly formatted code.

Python

2015-10-13T15:01:41Z

Lrogerson: /* Resources For Learning Python */

===Resources For Learning Python===

Though not comparable in size to the truly enormous userbase of C++, Java and C#, Python still boasts a highly active community and a correspondingly large number of resources. In the AG Euler group all users are encouraged to use Python 3. While some modules only exist for the older Python 2 codebase, in the overwhelming majority of cases they have been re-established in the Py3 framework (cf. [https://python3wos.appspot.com/ Python 3 Wall Of Superpowers] for current status of module transfer).

For the absolute beginner Codeacademy's free, interactive [https://www.codecademy.com/en/tracks/python learning module for Python] is a good starting point. Though it recommends approximately 13 hours for the course, those with a background in programming could probably finish it in under a day. For scientific computing it helps to have some familiarity with the Scipy stack, for which there is a nice introduction at [http://www.scipy-lectures.org/ scipy-lectures.org], particularly the Getting Started section.

Beyond that, it will be worth taking a look at the [https://web.stanford.edu/~mwaskom/software/seaborn/ Seaborn] module for attractive plotting functions. Those interested in Datajoint should also take a look at the Datajoint project page and that of [https://pythonhosted.org/neo/ Neo], the latter of which may be used for Python IO. Finally, if you are thinking of using Python for important code which will likely be reused, I would recommend reading the [https://www.python.org/dev/peps/pep-0008/ PEP0008 styleguide], to ensure you are writing correctly formatted code.

Python

2015-10-13T15:01:11Z

Lrogerson: Created page with "===Resources For Learning Python=== Though not comparable in size to the truly enormous userbase of C++, Java and C#, Python still boasts a highly active community and a corr..."

===Resources For Learning Python===

Though not comparable in size to the truly enormous userbase of C++, Java and C#, Python still boasts a highly active community and a correspondingly large number of resources. In the AG Euler group all users are encouraged to use Python 3. While some modules only exist for the older Python 2 codebase, in the overwhelming majority of cases they have been re-established in the Py3 framework (cf. [https://python3wos.appspot.com/ Python 3 Wall Of Superpowers] for current status of module transfer).

For the absolute beginner Codeacademy's free, interactive [https://www.codecademy.com/en/tracks/python learning module for Python] is a good starting point. Though it recommends approximately 13 hours for the course, those with a background in programming could probably finish it in under a day. For scientific computing it helps to have some familiarity with the Scipy stack, for which there is a nice introduction at [http://www.scipy-lectures.org/ scipy-lectures.org], particularly the Getting Started section.

Beyond that, it will be worth taking a look at the [https://web.stanford.edu/~mwaskom/software/seaborn/ Seaborn] module for attractive plotting functions. Those interested in Datajoint should also take a look at the Datajoint project page and that of Neo, the latter of which may be used for Python IO. Finally, if you are thinking of using Python for important code which will likely be reused, I would recommend reading the [https://www.python.org/dev/peps/pep-0008/ PEP0008 styleguide], to ensure you are writing correctly formatted code.

Main Page

2015-10-13T14:49:09Z

Lrogerson: /* Computational */

Welcome to the AG Euler wiki.
Feel free to contribute additional content.

==General==

[[Email Addresses]]

[[Internal Phone Numbers]]

==Experimental==

[[Ringer Solution]]

==Computational==

[[Cluster]]

[[Datajoint]]

[[Docker]]

[[Jupyter]]

Jupyter Notebooks

2015-10-13T14:48:55Z

Lrogerson: Lrogerson moved page Jupyter Notebooks to Jupyter

#REDIRECT [[Jupyter]]

Jupyter

2015-10-13T14:48:55Z

Lrogerson: Lrogerson moved page Jupyter Notebooks to Jupyter

==What are Jupyter Notebooks?==

Jupyter notebooks are browser based environments for editing, compiling and executing code. For a brief overview, see [http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 this Nature News article]. Until quite recently, these were developed exclusively for coding Python, although support has since expanded to accommodate a number of other languages. For a full list of language kernels, see [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages here]. Note that Igor Pro is not supported, and that many of these kernels are in alpha state and not suitable for daily use.

==Which notebook servers are currently running?==
We are currently running two Jupyter notebook servers in the CIN network, dubbed [https://en.wikipedia.org/wiki/Callisto_(moon) Callisto] and [https://en.wikipedia.org/wiki/Titan_(moon) Titan]. Callisto is geared towards stability and is the recommended environment for those who wish to write standard Python scripts. It comes with the [https://www.scipy.org/about.html scipy stack] preinstalled, which should be adequate for most scientific computing problems. The Callisto server will only be taken down after 6pm on Fridays, if at all, and should be functional again by the following morning.

Alongside this is a more experimental notebook server called Titan, which will be used for testing new computational tools including additional language kernels, parallelisation platforms such as Apache Spark, and GPU integration via CUDA. Titan will be down more frequently as new features are implemented and tested. If you have any features you would like to try out, or have implemented, let me know (luke.e.rogerson@gmail.com). If the implementations are sufficiently stable, we can migrate the content to Callisto at a later date.

==How do I access the AG Euler Jupyter Notebooks?==

To access the notebooks, you will have to specify the correct Putty configuration. A preconfigured file is available, and details for manual set up are given below.
If you forget the access details - each server is named after a moon, the year of its discovery is the localhost number, and the password is the planet which the moon orbits. If at any time you cannot remember the password or localhost address, refer to the wikipedia pages for the namesakes of the respective servers.

===Preconfigured Portable Putty===
[[File:AG Euler - Portable Putty - 13.10.2015.zip]]

===Manually Configure Putty===

#Open Putty, load your Cluster access configuration
#Under Category, click the '+' to expand ssh
#Click tunnels, and add the following:
##Destination 172.29.0.131:1610 .. Source: 1610
##Destination 172.29.0.131:1655 .. Source: 1655
#Save your configuration in the session tab
#Open the updated configuration by double clicking on it
#Use your CIN user account to log into the headnode
#Enter "ssh 'cn31'" to access the Euler group node where the servers are running
#Enter either https://localhost:1610 or https://localhost:1655 to access Callisto or Titan respectively. This should then take you to a login screen (or directly to the directories).
# Enter the password for the server (Case sensitive - Callisto: jupiter, Titan: saturn)

Cluster Access

2015-10-13T14:48:46Z

Lrogerson: Lrogerson moved page Cluster Access to Cluster

#REDIRECT [[Cluster]]

Cluster

2015-10-13T14:48:46Z

Lrogerson: Lrogerson moved page Cluster Access to Cluster

==What is the cluster?==

The CIN cluster is a High Performance Computing environment curated by the CIN IT services. It is composed of many 'nodes', which are computational environments which you can access both from your workstation and remotely, each of which can be used to run tasks independently. For computationally intensive work, it is recommended that you run your programs on this cluster. For the curious, the cluster is run on the [http://www.ch.cam.ac.uk/computing/maui-and-torque-introduction Maui/Torque architecture].

==How do I access the cluster?==

The cluster is accessed using a program called Putty. The [https://cinwiki/mediawiki/cinwiki/index.php/Storage_cluster instructions on the CIN wiki] should be sufficient for gaining access. In order to use the cluster, you will require a CIN user account, the name of which looks something like ''kelch'' for the user Karl Elch. It is possible to access the cluster remotely, and so execute code in the system from outside the CIN. To do so, follow the [https://cinwiki/mediawiki/cinwiki/index.php/VPN instructions on the CIN wiki]. Our nodes are 'cn30' and 'cn31', the latter of which is used for our Jupiter server. It is also possible to access GPU resources at 'gpu1', 'gpu2' and 'gpu3', but keep in mind that these are for general use by all groups in the CIN and may be engaged.

===Preconfigured Portable Putty===

Alternatively, you can access a preconfigured version of Putty intended for use with the Jupiter servers.

[[File:AG Euler - Portable Putty - 13.10.2015.zip]]

Ringer Solution

2015-10-13T14:11:18Z

Lrogerson:

{|class="wikitable"
|Chemical||2l||4l
|-
|NaCl||14.61||29.22
|-
|KCl||0.3728||0.7456
|-
|MgCL2.6H2O||0.406||0.812
|-
|NaH2PO4||0.3||0.6
|-
|Glucose||7.2||14.4
|-
|NaHCO3||4.369||8.738
|-
|CaCl2||0.588||1.176
|-
|L-Glut||0.146||0.292
|}

Jupyter

2015-10-13T13:39:33Z

Lrogerson: /* Manually Configure Putty */

==What are Jupyter Notebooks?==

Jupyter notebooks are browser based environments for editing, compiling and executing code. For a brief overview, see [http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 this Nature News article]. Until quite recently, these were developed exclusively for coding Python, although support has since expanded to accommodate a number of other languages. For a full list of language kernels, see [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages here]. Note that Igor Pro is not supported, and that many of these kernels are in alpha state and not suitable for daily use.

==Which notebook servers are currently running?==
We are currently running two Jupyter notebook servers in the CIN network, dubbed [https://en.wikipedia.org/wiki/Callisto_(moon) Callisto] and [https://en.wikipedia.org/wiki/Titan_(moon) Titan]. Callisto is geared towards stability and is the recommended environment for those who wish to write standard Python scripts. It comes with the [https://www.scipy.org/about.html scipy stack] preinstalled, which should be adequate for most scientific computing problems. The Callisto server will only be taken down after 6pm on Fridays, if at all, and should be functional again by the following morning.

Alongside this is a more experimental notebook server called Titan, which will be used for testing new computational tools including additional language kernels, parallelisation platforms such as Apache Spark, and GPU integration via CUDA. Titan will be down more frequently as new features are implemented and tested. If you have any features you would like to try out, or have implemented, let me know (luke.e.rogerson@gmail.com). If the implementations are sufficiently stable, we can migrate the content to Callisto at a later date.

==How do I access the AG Euler Jupyter Notebooks?==

To access the notebooks, you will have to specify the correct Putty configuration. A preconfigured file is available, and details for manual set up are given below.
If you forget the access details - each server is named after a moon, the year of its discovery is the localhost number, and the password is the planet which the moon orbits. If at any time you cannot remember the password or localhost address, refer to the wikipedia pages for the namesakes of the respective servers.

===Preconfigured Portable Putty===
[[File:AG Euler - Portable Putty - 13.10.2015.zip]]

===Manually Configure Putty===

#Open Putty, load your Cluster access configuration
#Under Category, click the '+' to expand ssh
#Click tunnels, and add the following:
##Destination 172.29.0.131:1610 .. Source: 1610
##Destination 172.29.0.131:1655 .. Source: 1655
#Save your configuration in the session tab
#Open the updated configuration by double clicking on it
#Use your CIN user account to log into the headnode
#Enter "ssh 'cn31'" to access the Euler group node where the servers are running
#Enter either https://localhost:1610 or https://localhost:1655 to access Callisto or Titan respectively. This should then take you to a login screen (or directly to the directories).
# Enter the password for the server (Case sensitive - Callisto: jupiter, Titan: saturn)

Docker (deprecated)

2015-10-13T13:36:05Z

Lrogerson: /* What is docker? */

==What is docker?==

Docker is a platform for running virtual machines called ''containers''. These are functionally analogous to the operating systems on a computer, though you can run them from within your operating system. They are useful because they create lightweight, customisable environments within which you can run software. This is particularly helpful when you have programs which have very particular dependencies, or require a linux base installation in order to function correctly. You can read more about Docker [https://www.docker.com/whatisdocker here].

==What do we use docker for?==

The Euler group is currently using docker to host [[Jupyter_Notebooks|our two Jupyter notebook servers]], though in the past we have also used it for hosting a MySQL server. To see what containers are currently running, ssh into 'cn31' and enter the command: <blockquote>docker ps</blockquote> This will return a list of the currently active docker containers, the images they are built from, how long they have been online and what ports they are forwarded to.

==How do I build my own docker containers?==
New docker containers can be built using the command: <blockquote>docker run [...]</blockquote> For example, to build a new scipyserver called 'miranda', forwarded to port 1948 on localhost with the password 'uranus', you would enter: <blockquote>docker run -d -p 1948:8888 --name miranda -e "PASSWORD=uranus" -e DISPLAY=$DISPLAY ipython/scipyserver</blockquote> Each of the -tags has a unique function in building the server. The last entry specifies the image from which the container is built, in this case an ipython server. You can find more images over at [https://hub.docker.com/ Dockerhub].

<blockquote>
{|class="wikitable"
|'''Tag'''||'''Function'''
|-
| -d||Run in the background
|-
| -p 1948:8888||Forward port 8888 to 1948
|-
| --name miranda||Specify container name as miranda
|-
| -e "PASSWORD=uranus"||Set password to uranus
|-
| -e DISPLAY=$DISPLAY||Needed for plotting in Jupyter
|-
| --help||Show different tag options for given command
|-
| -v||Mount volume, see below
|} </blockquote>

Data stored in docker containers is transient and is lost when the container is deleted. In order to set the containers so that they can access volumes on the host system and persist their content after they are deleted, we have to use additional tags to mount directories in the container to directories on the host system. Here is an example of this for another ipython server:<blockquote>docker run -v ~/Notebooks:/notebooks/Notebooks -v ~/Data:/notebooks/Data:ro -v /tmp/.X11-unix:/tmp/.X11-unix:ro ipython/scipyserver</blockquote> Here, three different directories have been mounted: a volume which the container can write to, under notebooks/Notebooks, a volume which the container can read, under notebooks/Data (the :ro tag specifies read only), and a further folder used for X11 forwarding, which is necessary for getting figures in the Jupiter server.

For a more thorough walkthough, see the official [https://docs.docker.com/linux/started/ Getting Started] documentation.

Cluster

2015-10-13T13:34:12Z

Lrogerson: /* Preconfigured Portable Putty */

==What is the cluster?==

The CIN cluster is a High Performance Computing environment curated by the CIN IT services. It is composed of many 'nodes', which are computational environments which you can access both from your workstation and remotely, each of which can be used to run tasks independently. For computationally intensive work, it is recommended that you run your programs on this cluster. For the curious, the cluster is run on the [http://www.ch.cam.ac.uk/computing/maui-and-torque-introduction Maui/Torque architecture].

==How do I access the cluster?==

The cluster is accessed using a program called Putty. The [https://cinwiki/mediawiki/cinwiki/index.php/Storage_cluster instructions on the CIN wiki] should be sufficient for gaining access. In order to use the cluster, you will require a CIN user account, the name of which looks something like ''kelch'' for the user Karl Elch. It is possible to access the cluster remotely, and so execute code in the system from outside the CIN. To do so, follow the [https://cinwiki/mediawiki/cinwiki/index.php/VPN instructions on the CIN wiki]. Our nodes are 'cn30' and 'cn31', the latter of which is used for our Jupiter server. It is also possible to access GPU resources at 'gpu1', 'gpu2' and 'gpu3', but keep in mind that these are for general use by all groups in the CIN and may be engaged.

===Preconfigured Portable Putty===

Alternatively, you can access a preconfigured version of Putty intended for use with the Jupiter servers.

[[File:AG Euler - Portable Putty - 13.10.2015.zip]]

Cluster

2015-10-13T13:33:58Z

Lrogerson: /* How do I access the cluster? */

==What is the cluster?==

The CIN cluster is a High Performance Computing environment curated by the CIN IT services. It is composed of many 'nodes', which are computational environments which you can access both from your workstation and remotely, each of which can be used to run tasks independently. For computationally intensive work, it is recommended that you run your programs on this cluster. For the curious, the cluster is run on the [http://www.ch.cam.ac.uk/computing/maui-and-torque-introduction Maui/Torque architecture].

==How do I access the cluster?==

The cluster is accessed using a program called Putty. The [https://cinwiki/mediawiki/cinwiki/index.php/Storage_cluster instructions on the CIN wiki] should be sufficient for gaining access. In order to use the cluster, you will require a CIN user account, the name of which looks something like ''kelch'' for the user Karl Elch. It is possible to access the cluster remotely, and so execute code in the system from outside the CIN. To do so, follow the [https://cinwiki/mediawiki/cinwiki/index.php/VPN instructions on the CIN wiki]. Our nodes are 'cn30' and 'cn31', the latter of which is used for our Jupiter server. It is also possible to access GPU resources at 'gpu1', 'gpu2' and 'gpu3', but keep in mind that these are for general use by all groups in the CIN and may be engaged.

===Preconfigured Portable Putty===

Alternatively, you can access a preconfigured version of Putty intended for use with the Jupiter servers.

File:AG Euler - Portable Putty - 13.10.2015.zip

Jupyter

2015-10-13T12:07:02Z

Lrogerson: /* How do I access the AG Euler Jupyter Notebooks? */

==What are Jupyter Notebooks?==

Jupyter notebooks are browser based environments for editing, compiling and executing code. For a brief overview, see [http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 this Nature News article]. Until quite recently, these were developed exclusively for coding Python, although support has since expanded to accommodate a number of other languages. For a full list of language kernels, see [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages here]. Note that Igor Pro is not supported, and that many of these kernels are in alpha state and not suitable for daily use.

==Which notebook servers are currently running?==
We are currently running two Jupyter notebook servers in the CIN network, dubbed [https://en.wikipedia.org/wiki/Callisto_(moon) Callisto] and [https://en.wikipedia.org/wiki/Titan_(moon) Titan]. Callisto is geared towards stability and is the recommended environment for those who wish to write standard Python scripts. It comes with the [https://www.scipy.org/about.html scipy stack] preinstalled, which should be adequate for most scientific computing problems. The Callisto server will only be taken down after 6pm on Fridays, if at all, and should be functional again by the following morning.

Alongside this is a more experimental notebook server called Titan, which will be used for testing new computational tools including additional language kernels, parallelisation platforms such as Apache Spark, and GPU integration via CUDA. Titan will be down more frequently as new features are implemented and tested. If you have any features you would like to try out, or have implemented, let me know (luke.e.rogerson@gmail.com). If the implementations are sufficiently stable, we can migrate the content to Callisto at a later date.

==How do I access the AG Euler Jupyter Notebooks?==

To access the notebooks, you will have to specify the correct Putty configuration. A preconfigured file is available, and details for manual set up are given below.
If you forget the access details - each server is named after a moon, the year of its discovery is the localhost number, and the password is the planet which the moon orbits. If at any time you cannot remember the password or localhost address, refer to the wikipedia pages for the namesakes of the respective servers.

===Preconfigured Portable Putty===
[[File:AG Euler - Portable Putty - 13.10.2015.zip]]

===Manually Configure Putty===

#Open Putty, load your Cluster access configuration
#Under Category, click the '+' to expand ssh
#Click tunnels, and add the following:
##Destination 172.29.0.131:1610 .. Source: 1610
##Destination 172.29.0.131:1655 .. Source: 1655
#Save your configuration in the session tab
#Open the updated configuration by double clicking on it
#Use your CIN user account to log into the headnode
#Enter "ssh 'cn31'" to access the Euler group node where the servers are running
#Enter either https://localhost:1610 or https://localhost:1655 to access Callisto or Titan respectively. This should then take you to a login screen (or directly to the directories).

v. Enter the password for the server (Case sensitive - Callisto: jupiter, Titan: saturn)

Jupyter

2015-10-13T12:06:32Z

Lrogerson: /* Manually Configure Putty */

==What are Jupyter Notebooks?==

Jupyter notebooks are browser based environments for editing, compiling and executing code. For a brief overview, see [http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 this Nature News article]. Until quite recently, these were developed exclusively for coding Python, although support has since expanded to accommodate a number of other languages. For a full list of language kernels, see [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages here]. Note that Igor Pro is not supported, and that many of these kernels are in alpha state and not suitable for daily use.

==Which notebook servers are currently running?==
We are currently running two Jupyter notebook servers in the CIN network, dubbed [https://en.wikipedia.org/wiki/Callisto_(moon) Callisto] and [https://en.wikipedia.org/wiki/Titan_(moon) Titan]. Callisto is geared towards stability and is the recommended environment for those who wish to write standard Python scripts. It comes with the [https://www.scipy.org/about.html scipy stack] preinstalled, which should be adequate for most scientific computing problems. The Callisto server will only be taken down after 6pm on Fridays, if at all, and should be functional again by the following morning.

Alongside this is a more experimental notebook server called Titan, which will be used for testing new computational tools including additional language kernels, parallelisation platforms such as Apache Spark, and GPU integration via CUDA. Titan will be down more frequently as new features are implemented and tested. If you have any features you would like to try out, or have implemented, let me know (luke.e.rogerson@gmail.com). If the implementations are sufficiently stable, we can migrate the content to Callisto at a later date.

==How do I access the AG Euler Jupyter Notebooks?==

To access the notebooks, you will have to specify the correct Putty configuration. A preconfigured file is available, and details for manual set up are given below.

===Preconfigured Portable Putty===
[[File:AG Euler - Portable Putty - 13.10.2015.zip]]

===Manually Configure Putty===

#Open Putty, load your Cluster access configuration
#Under Category, click the '+' to expand ssh
#Click tunnels, and add the following:
##Destination 172.29.0.131:1610 .. Source: 1610
##Destination 172.29.0.131:1655 .. Source: 1655
#Save your configuration in the session tab
#Open the updated configuration by double clicking on it
#Use your CIN user account to log into the headnode
#Enter "ssh 'cn31'" to access the Euler group node where the servers are running
#Enter either https://localhost:1610 or https://localhost:1655 to access Callisto or Titan respectively. This should then take you to a login screen (or directly to the directories).

v. Enter the password for the server (Case sensitive - Callisto: jupiter, Titan: saturn)

Jupyter

2015-10-13T12:06:11Z

Lrogerson: /* How do I access the AG Euler Jupyter Notebooks? */

==What are Jupyter Notebooks?==

Jupyter notebooks are browser based environments for editing, compiling and executing code. For a brief overview, see [http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 this Nature News article]. Until quite recently, these were developed exclusively for coding Python, although support has since expanded to accommodate a number of other languages. For a full list of language kernels, see [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages here]. Note that Igor Pro is not supported, and that many of these kernels are in alpha state and not suitable for daily use.

==Which notebook servers are currently running?==
We are currently running two Jupyter notebook servers in the CIN network, dubbed [https://en.wikipedia.org/wiki/Callisto_(moon) Callisto] and [https://en.wikipedia.org/wiki/Titan_(moon) Titan]. Callisto is geared towards stability and is the recommended environment for those who wish to write standard Python scripts. It comes with the [https://www.scipy.org/about.html scipy stack] preinstalled, which should be adequate for most scientific computing problems. The Callisto server will only be taken down after 6pm on Fridays, if at all, and should be functional again by the following morning.

Alongside this is a more experimental notebook server called Titan, which will be used for testing new computational tools including additional language kernels, parallelisation platforms such as Apache Spark, and GPU integration via CUDA. Titan will be down more frequently as new features are implemented and tested. If you have any features you would like to try out, or have implemented, let me know (luke.e.rogerson@gmail.com). If the implementations are sufficiently stable, we can migrate the content to Callisto at a later date.

==How do I access the AG Euler Jupyter Notebooks?==

To access the notebooks, you will have to specify the correct Putty configuration. A preconfigured file is available, and details for manual set up are given below.

===Preconfigured Portable Putty===
[[File:AG Euler - Portable Putty - 13.10.2015.zip]]

===Manually Configure Putty===

#Open Putty, load your Cluster access configuration
#Under Category, click the '+' to expand ssh
#Click tunnels, and add the following:
##Destination 172.29.0.131:1610 .. Source: 1610
##Destination 172.29.0.131:1655 .. Source: 1655
#Save your configuration in the session tab
#Open the updated configuration by double clicking on it
#Use your CIN user account to log into the headnode
#Enter "ssh 'cn31'" to access the Euler group node where the servers are running
#Enter either https://localhost:1610 or https://localhost:1655 to access Callisto or Titan respectively. This should then take you to a login screen (or directly to the directories).

v. Enter the password for the server (Case sensitive - Callisto: jupiter, Titan: saturn)

Each server is named after a moon, the year of its discovery is the localhost number, and the password is the planet which the moon orbits. If at any time you cannot remember the password or localhost address, refer to the wikipedia page for the namesakes of the respective servers.

Jupyter

2015-10-13T12:05:14Z

Lrogerson: /* How do I access the AG Euler Jupyter Notebooks? */

==What are Jupyter Notebooks?==

Jupyter notebooks are browser based environments for editing, compiling and executing code. For a brief overview, see [http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 this Nature News article]. Until quite recently, these were developed exclusively for coding Python, although support has since expanded to accommodate a number of other languages. For a full list of language kernels, see [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages here]. Note that Igor Pro is not supported, and that many of these kernels are in alpha state and not suitable for daily use.

==Which notebook servers are currently running?==
We are currently running two Jupyter notebook servers in the CIN network, dubbed [https://en.wikipedia.org/wiki/Callisto_(moon) Callisto] and [https://en.wikipedia.org/wiki/Titan_(moon) Titan]. Callisto is geared towards stability and is the recommended environment for those who wish to write standard Python scripts. It comes with the [https://www.scipy.org/about.html scipy stack] preinstalled, which should be adequate for most scientific computing problems. The Callisto server will only be taken down after 6pm on Fridays, if at all, and should be functional again by the following morning.

Alongside this is a more experimental notebook server called Titan, which will be used for testing new computational tools including additional language kernels, parallelisation platforms such as Apache Spark, and GPU integration via CUDA. Titan will be down more frequently as new features are implemented and tested. If you have any features you would like to try out, or have implemented, let me know (luke.e.rogerson@gmail.com). If the implementations are sufficiently stable, we can migrate the content to Callisto at a later date.

==How do I access the AG Euler Jupyter Notebooks?==

To access the notebooks, you will have to specify the correct Putty configuration. This has been predefined in the Portable Putty file, linked at the bottom of the page. To access the servers manually you need to do the following:

#Open Putty, load your Cluster access configuration
#Under Category, click the '+' to expand ssh
#Click tunnels, and add the following:
##Destination 172.29.0.131:1610 .. Source: 1610
##Destination 172.29.0.131:1655 .. Source: 1655
#Save your configuration in the session tab
#Open the updated configuration by double clicking on it
#Use your CIN user account to log into the headnode
#Enter "ssh 'cn31'" to access the Euler group node where the servers are running
#Enter either https://localhost:1610 or https://localhost:1655 to access Callisto or Titan respectively. This should then take you to a login screen (or directly to the directories).

v. Enter the password for the server (Case sensitive - Callisto: jupiter, Titan: saturn)

Each server is named after a moon, the year of its discovery is the localhost number, and the password is the planet which the moon orbits. If at any time you cannot remember the password or localhost address, refer to the wikipedia page for the namesakes of the respective servers.

===Preconfigured Portable Putty===
[[File:AG Euler - Portable Putty - 13.10.2015.zip]]

Jupyter

2015-10-13T12:04:00Z

Lrogerson: /* How do I access the AG Euler Jupyter Notebooks? */

==What are Jupyter Notebooks?==

Jupyter notebooks are browser based environments for editing, compiling and executing code. For a brief overview, see [http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 this Nature News article]. Until quite recently, these were developed exclusively for coding Python, although support has since expanded to accommodate a number of other languages. For a full list of language kernels, see [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages here]. Note that Igor Pro is not supported, and that many of these kernels are in alpha state and not suitable for daily use.

==Which notebook servers are currently running?==
We are currently running two Jupyter notebook servers in the CIN network, dubbed [https://en.wikipedia.org/wiki/Callisto_(moon) Callisto] and [https://en.wikipedia.org/wiki/Titan_(moon) Titan]. Callisto is geared towards stability and is the recommended environment for those who wish to write standard Python scripts. It comes with the [https://www.scipy.org/about.html scipy stack] preinstalled, which should be adequate for most scientific computing problems. The Callisto server will only be taken down after 6pm on Fridays, if at all, and should be functional again by the following morning.

Alongside this is a more experimental notebook server called Titan, which will be used for testing new computational tools including additional language kernels, parallelisation platforms such as Apache Spark, and GPU integration via CUDA. Titan will be down more frequently as new features are implemented and tested. If you have any features you would like to try out, or have implemented, let me know (luke.e.rogerson@gmail.com). If the implementations are sufficiently stable, we can migrate the content to Callisto at a later date.

==How do I access the AG Euler Jupyter Notebooks?==

To access the notebooks, you will have to specify the correct Putty configuration. This has been predefined in the Portable Putty file that you can download from [[File:AG Euler - Portable Putty - 13.10.2015.zip || here]]. To access the servers manually you need to do the following:

#Open Putty, load your Cluster access configuration
#Under Category, click the '+' to expand ssh
#Click tunnels, and add the following:
##Destination 172.29.0.131:1610 .. Source: 1610
##Destination 172.29.0.131:1655 .. Source: 1655
#Save your configuration in the session tab
#Open the updated configuration by double clicking on it
#Use your CIN user account to log into the headnode
#Enter "ssh 'cn31'" to access the Euler group node where the servers are running
#Enter either https://localhost:1610 or https://localhost:1655 to access Callisto or Titan respectively. This should then take you to a login screen (or directly to the directories).

v. Enter the password for the server (Case sensitive - Callisto: jupiter, Titan: saturn)

Each server is named after a moon, the year of its discovery is the localhost number, and the password is the planet which the moon orbits. If at any time you cannot remember the password or localhost address, refer to the wikipedia page for the namesakes of the respective servers.

Jupyter

2015-10-13T12:03:28Z

Lrogerson: /* How do I access the AG Euler Jupyter Notebooks? */

==What are Jupyter Notebooks?==

Jupyter notebooks are browser based environments for editing, compiling and executing code. For a brief overview, see [http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 this Nature News article]. Until quite recently, these were developed exclusively for coding Python, although support has since expanded to accommodate a number of other languages. For a full list of language kernels, see [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages here]. Note that Igor Pro is not supported, and that many of these kernels are in alpha state and not suitable for daily use.

==Which notebook servers are currently running?==
We are currently running two Jupyter notebook servers in the CIN network, dubbed [https://en.wikipedia.org/wiki/Callisto_(moon) Callisto] and [https://en.wikipedia.org/wiki/Titan_(moon) Titan]. Callisto is geared towards stability and is the recommended environment for those who wish to write standard Python scripts. It comes with the [https://www.scipy.org/about.html scipy stack] preinstalled, which should be adequate for most scientific computing problems. The Callisto server will only be taken down after 6pm on Fridays, if at all, and should be functional again by the following morning.

Alongside this is a more experimental notebook server called Titan, which will be used for testing new computational tools including additional language kernels, parallelisation platforms such as Apache Spark, and GPU integration via CUDA. Titan will be down more frequently as new features are implemented and tested. If you have any features you would like to try out, or have implemented, let me know (luke.e.rogerson@gmail.com). If the implementations are sufficiently stable, we can migrate the content to Callisto at a later date.

==How do I access the AG Euler Jupyter Notebooks?==

To access the notebooks, you will have to specify the correct Putty configuration. This has been predefined in the Portable Putty file that you can download from [[File:AG Euler - Portable Putty - 13.10.2015.zip | here]]. To access the servers manually you need to do the following:

#Open Putty, load your Cluster access configuration
#Under Category, click the '+' to expand ssh
#Click tunnels, and add the following:
##Destination 172.29.0.131:1610 .. Source: 1610
##Destination 172.29.0.131:1655 .. Source: 1655
#Save your configuration in the session tab
#Open the updated configuration by double clicking on it
#Use your CIN user account to log into the headnode
#Enter "ssh 'cn31'" to access the Euler group node where the servers are running
#Enter either https://localhost:1610 or https://localhost:1655 to access Callisto or Titan respectively. This should then take you to a login screen (or directly to the directories).

v. Enter the password for the server (Case sensitive - Callisto: jupiter, Titan: saturn)

Each server is named after a moon, the year of its discovery is the localhost number, and the password is the planet which the moon orbits. If at any time you cannot remember the password or localhost address, refer to the wikipedia page for the namesakes of the respective servers.

File:AG Euler - Portable Putty - 13.10.2015.zip

2015-10-13T12:00:18Z

Lrogerson: Preconfigured Putty file which can be used to access the CIN cluster and Jupyter notebooks. Requires CIN username and password. Usage: Select the clusterTunnel configuration then click open. This will open a terminal which will connect to the CIN clu...

Preconfigured Putty file which can be used to access the CIN cluster and Jupyter notebooks. Requires CIN username and password.

Usage: Select the clusterTunnel configuration then click open. This will open a terminal which will connect to the CIN cluster directly if you are within the CIN or connected to the CIN via openVPN. You will then be directed to enter your username and password. Once you are in, type 'ssh cn31' to enter the 'cn31' node.

Main Page

2015-10-13T11:56:40Z

Lrogerson:

Welcome to the AG Euler wiki.
Feel free to contribute additional content.

==General==

[[Email Addresses]]

[[Internal Phone Numbers]]

==Experimental==

[[Ringer Solution]]

==Computational==

[[Cluster Access]]

[[Datajoint]]

[[Docker]]

[[Jupyter Notebooks]]

Main Page

2015-10-13T11:56:21Z

Lrogerson: /* Technical */

Welcome to the AG Euler wiki.
Feel free to contribute additional content.

==General==

[[Email Addresses]]

[[Internal Phone Numbers]]

==Computational==

[[Cluster Access]]

[[Datajoint]]

[[Docker]]

[[Jupyter Notebooks]]

Ringer Solution

2015-10-13T11:55:51Z

Lrogerson: /* Recipe */

{|class="wikitable"
|Chemical||2l||4l
|-
|NaCl||14.61||29.22
|-
|KCl||0.3728||0.7456
|-
|MgCL26H2O||0.406||0.812
|-
|NaH2PO4||0.3||0.6
|-
|Glucose||7.2||14.4
|-
|NaHCO3||4.369||8.738
|-
|CaCl2||0.588||1.176
|-
|L-Glut||0.146||0.292
|}

Ringer Solution

2015-10-13T11:55:37Z

Lrogerson: /* Recipe */

==Recipe==

{|class="wikitable"
|Chemical||2l||4l
|-
|NaCl||14.61||29.22
|-
|KCl||0.3728||0.7456
|-
|MgCL26H2O||0.406||0.812
|-
|NaH2PO4||0.3||0.6
|-
|Glucose||7.2||14.4
|-
|NaHCO3||4.369||8.738
|-
|CaCl2||0.588||1.176
|-
|L-Glut||0.146||0.292
|}

Ringer Solution

2015-10-13T11:55:06Z

Lrogerson: /* Recipe */

==Recipe==

{|class="wikitable"
|Chemical||2l||4l
|-
|NaCl||14.61||29.22
|-
|KCl||0.3728||0.7456
|-
|MgCL26H{{sub|2}}O||0.406||0.812
|-
|NaH{{sub|2}}PO{{sub|4}}||0.3||0.6
|-
|Glucose||7.2||14.4
|-
|NaHCO{{sub|3}}||4.369||8.738
|-
|CaCl{{sub|2}}||0.588||1.176
|-
|L-Glut||0.146||0.292
|}

Ringer Solution

2015-10-13T11:53:08Z

Lrogerson: /* Recipe */

==Recipe==

{|class="wikitable"
|Chemical||2l||4l
|-
|NaCl||14.61||29.22
|-
|KCl||0.3728||0.7456
|-
|MgCL{{sub|2}}6H{{sub|2}}O||0.406||0.812
|-
|NaH{{sub|2}}PO{{sub|4}}||0.3||0.6
|-
|Glucose||7.2||14.4
|-
|NaHCO{{sub|3}}||4.369||8.738
|-
|CaCl{{sub|2}}||0.588||1.176
|-
|L-Glut||0.146||0.292
|}

Ringer Solution

2015-10-13T11:52:57Z

Lrogerson: /* Recipe */

==Recipe==

{|class="wikitable"
|Chemical||2l||4l
|-
|NaCl||14.61||29.22
|-
|KCl||0.3728||0.7456
|-
|MgCL{sub|2}~6H{{sub|2}}O||0.406||0.812
|-
|NaH{{sub|2}}PO{{sub|4}}||0.3||0.6
|-
|Glucose||7.2||14.4
|-
|NaHCO{{sub|3}}||4.369||8.738
|-
|CaCl{{sub|2}}||0.588||1.176
|-
|L-Glut||0.146||0.292
|}

Ringer Solution

2015-10-13T11:52:24Z

Lrogerson: /* Recipe */

==Recipe==

{|class="wikitable"
|Chemical||2l||4l
|-
|NaCl||14.61||29.22
|-
|KCl||0.3728||0.7456
|-
|MgCL~2~6H{{sub|2}}O||0.406||0.812
|-
|NaH{{sub|2}}PO{{sub|4}}||0.3||0.6
|-
|Glucose||7.2||14.4
|-
|NaHCO{{sub|3}}||4.369||8.738
|-
|CaCl{{sub|2}}||0.588||1.176
|-
|L-Glut||0.146||0.292
|}

Ringer Solution

2015-10-13T11:49:58Z

Lrogerson: Created page with "==Recipe== {|class="wikitable" |Chemical||2l||4l |- |NaCl||14.61||29.22 |- |KCl||0.3728||0.7456 |- |MgCL{{sub|2}}6H{{sub|2}}O||0.406||0.812 |- |NaH{{sub|2}}PO{{sub|4}}||0.3||..."

Datajoint

2015-10-13T10:48:21Z

Lrogerson: /* What is Datajoint? */

==What is Datajoint?==
There is an initiative in the Euler lab group to change the way our data is stored and accessed. Currently, data is stored in each lab members directories with little consistency in naming and directory organisation between individuals. The ambition of the Euler lab Datajoint project is to move from the present system to one where data is processed and stored in a consistent manner which is easily accessible to everyone.

In order to achieve this, a pipeline of computational tools will need to be established and integrated into the experimental and analytical workflow in the lab. The development is being coordinated by Prof. Dr. Thomas Euler, Dr. Philipp Berens, Dr. Tom Baden, Luke Rogerson, and Theresa Stadler, with additional contributions from the CIN IT services and other members of AG Euler and AG Bethge. If you have any questions, please refer them to one of the individuals listed above.

Integral to this process is the use of the [https://datajoint.github.io/ Datajoint toolkit]. In the words of the designers:
<blockquote>DataJoint for MATLAB and Python3 is a high-level programming interface for MySQL databases to support data processing chains in science labs. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, and querying data.</blockquote>

Datajoint

2015-10-13T10:48:04Z

Lrogerson: /* What is Datajoint? */

==What is Datajoint?==
There is an initiative in the Euler lab group to change the way our data is stored and accessed. Currently, data is stored in each lab members directories with little consistency in naming and directory organisation between individuals. The ambition of the Euler lab Datajoint project is to move from the present system to one where data is processed and stored in a consistent manner which is easily accessible to everyone.

Integral to this process is the use of the [https://datajoint.github.io/ Datajoint toolkit]. In the words of the designers:
<blockquote>DataJoint for MATLAB and Python3 is a high-level programming interface for MySQL databases to support data processing chains in science labs. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, and querying data.</blockquote>

In order to achieve this, a pipeline of computational tools will need to be established and integrated into the experimental and analytical workflow in the lab. The development is being coordinated by Prof. Dr. Thomas Euler, Dr. Philipp Berens, Dr. Tom Baden, Luke Rogerson, and Theresa Stadler, with additional contributions from the CIN IT services and other members of AG Euler and AG Bethge. If you have any questions, please refer them to one of the individuals listed above.

Datajoint

2015-10-13T10:47:02Z

Lrogerson: Created page with "==What is Datajoint?== In the words of the designers: <blockquote>DataJoint for MATLAB and Python3 is a high-level programming interface for MySQL databases to support data p..."

==What is Datajoint?==

In the words of the designers:
<blockquote>DataJoint for MATLAB and Python3 is a high-level programming interface for MySQL databases to support data processing chains in science labs. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, and querying data.</blockquote>

There is an initiative in the Euler lab group to change the way our data is stored and accessed. Currently, data is stored in each lab members directories with little consistency in naming and directory organisation between individuals. The ambition of the Euler lab Datajoint project is to move from the present system to one where data is processed and stored in a consistent manner which is easily accessible to everyone.

In order to achieve this, a pipeline of computational tools will need to be established and integrated into the experimental and analytical workflow in the lab. The development is being coordinated by Prof. Dr. Thomas Euler, Dr. Philipp Berens, Dr. Tom Baden, Luke Rogerson, and Theresa Stadler, with additional contributions from the CIN IT services and other members of AG Euler and AG Bethge. If you have any questions, please refer them to one of the individuals listed above.