How to Cross Compile Python and Run in Embedded System

How To Cross Compile Python and Run in Embedded System by cawan (cawan[at]ieee.org or chuiyewleong[at]hotmail.com) on 1/12/2012 The cross compilation process of python 2.7.1 is very straight forward. There is a simple guide about it in the following link, http://www.cnx-software.com/2011/02/04/cross-compiling http://www.cnx-software.c om/2011/02/04/cross-compiling-python-for-mips-and-arm-pla -python-for-mips-and-arm-platforms/ tforms/ Let us do the same thing and wish to get the same result. The embedded platform to be used is MIPS32 24k. First of all, let us download the source of python 2.7.1. Host:# wget http://www.python.org/ftp/py http://www.python.org/ftp/python/2.7.1/Python-2.7.1.tgz thon/2.7.1/Python-2.7.1.tgz Then get the patch accordingly. Host:# wget http://www.cnx-software.com/ http://www.cnx-software.com/patch/python-2.7.1-cross-com patch/python-2.7.1-cross-compile.patch pile.patch So, we have 2 files now. Host:# ls python-2.7.1-cross-compile.patch

Python-2.7.1.tgz

Let us start. Host:# tar xzvf Python-2.7.1.tgz ... ... Host:# ls Python-2.7.1 python-2.7.1-cross-compile.patch Host:# cd Python-2.7.1

Python-2.7.1.tgz

Since we need python and pgen as assisting tools for our cross compilation process, so we need to make a copy of them to be run in our host machine. Host:# ... ... Host:# ... ... Host:# Host:#

./configure

make python Parser/pgen

mv python hostpython mv Parser/pgen Parser/hostpgen

Well, we can later make use of the hostpython and hostpgen in our following cross compilation process. Before that, let us revert the source to original and get ready to cross compilation. Host:# make distclean ... ... Host:# Let us apply the patch first. Host:# patch -p1 < ../python-2.7.1-cross-com ../python-2.7.1-cross-compile.patch pile.patch (Stripping trailing CRs from patch.) patching file configure (Stripping trailing CRs from patch.) patching file configure.in (Stripping trailing CRs from patch.) patching file Makefile.pre.in (Stripping trailing CRs from patch.) patching file setup.py Host:# Good, it seems the patch is fine. Now let us run the configure tool to generate appropriate makefile.

Host:# ./configure --host=mips-linux-gnu --target=mips-linux-gnu \ CC="mips-linux-gnu-gcc -EL" CXX="mips-linux-gnu-g++ -EL" AR=mips-linux-gnu-ar \ RANLIB=mips-linux-gnu-ranlib We made some modification to the command line from the original one because the cross compiler that we are using is different to the reference. For the same reason, we will make some modification to the make command also, and run it accordingly. Before that, let us make sure the variables BLDSHARED and HOSTPYTHON are really being used in the make process. Host:# grep BLDSHARED Makefile BLDSHARED= $(CC) -shared $(LDFLAGS) ... ... Host:# grep HOSTPYTHON Makefile HOSTPYTHON= ./$(BUILDPYTHON) ... ... Host:# Good, the results are really made sense. Let us start the make process now. Host:# make HOSTPYTHON=./hostpython HOSTPGEN=./Parser/hostpgen \ BLDSHARED="mips-linux-gnu-gcc BLDSHARED="mips-linux-gnu -gcc -EL -shared" CROSS_COMPILE=mips-linux-gnu- \ CROSS_COMPILE_TARGET=yes However, we can't get the job done as what the reference did, and the error message is stated as below. File "./setup.py", line 316 self.announce('*** WARNING: renaming "%s" since importing it' ^ IndentationError: expected an indented block make: *** [sharedmods] Error 1 Host:# It seems this is just a simple indentation error of setup.py. Let us open the setup.py and add some spaces as indentation accordingly. if os.environ.get('CROSS_COMPILE_TARGET') != "yes": self.announce('*** WARNING: renaming "%s" since importing it' ' failed: %s' % (ext.name, why), level=3) After the modification, it supposes to look like this. if os.environ.get('CROSS_COMPILE_TARGET') != "yes": self.announce('*** WARNING: renaming "%s" since importing it' ' failed: %s' % (ext.name, why), level=3) Now, let us rerun the make process. After a while, the make process completed. So, let us perform make install now. Host:# make install HOSTPYTHON=./hostpython BLDSHARED="mips-linux-gnu-gcc -EL -shared" \ CROSS_COMPILE=mips-linux-gnuCROSS_COMPILE=mips-linuxgnu- CROSS_COMPILE_TARGET=yes DESTDIR=$PWD/cawanpython However, we get error again, and the error message is stated as below. File "/home/smp383/mips-4.3/bi "/home/smp383/mips-4.3/bin/testpython/Python-2.7.1/ca n/testpython/Python-2.7.1/cawanpython/usr/local/lib/ wanpython/usr/local/lib/ python2.7/struct.py", line 1, in from _struct import * ImportError: /home/smp383/mips-4.3/bin/t /home/smp383/mips-4.3/bin/testpython/Python-2.7.1/build estpython/Python-2.7.1/build/lib.linux-i686-2.7/ /lib.linux-i686-2.7/  _struct.so: cannot open shared shared object file: No such file file or directory make: *** [libinstall] Error 1 Host:# Well, it seems the _struct.so can't be found. Let us check the shared library is available or not. Host:# ls -l ./build/lib.linux-i686-2.7/_struct.so

-rwxr-xr-x 1 root root 85558 2012-11-29 17:34 ./build/lib.linux-i686-2.7/_struct.so Host:# Strange, the _struct.so is in the path. Anyway, let us check the destination directory. Host:# cd ./cawanpython/usr/local/l ./cawanpython/usr/local/lib/python2.7/ ib/python2.7/ Host:# ls lib lib2to3/ lib-old/ lib-tk/ Host:# ls lib Unfortunately, the lib-dynload directory is even not generated yet. Let us try to ignore the make error and let the make process to continue, and then observe the destination directory again. Host:# make -i install HOSTPYTHON=./hostpython BLDSHARED="mips-linux-gnu-gcc BLDSHARED="mips-linux-gnu-gcc -EL -shared" \ CROSS_COMPILE=mips-linux-gnuCROSS_COMPILE=mips-linuxgnu- CROSS_COMPILE_TARGET=yes DESTDIR=$PWD/cawanpython Please note that the -i option is added after the make command. After a while, the make process done. Let's check the lib-dynload directory. Host:# cd ./cawanpython/usr/local/ bin/ include/ lib/ share/ Host:# cd ./cawanpython/usr/local/l ./cawanpython/usr/local/lib/python2.7/ ib/python2.7/ Host:# ls liblib-dynload/ lib-old/ lib-tk/ Host:# ls lib-dynload/ array.so _ctypes.so linuxaudiodev.so audioop.so _ctypes_test.so _locale.so  _bisect.so datetime.so _lsprof.so cmath.so dl.so math.so  _codecs_cn.so _elementtree.so _md5.so  _codecs_hk.so fcntl.so mmap.so  _codecs_iso2022.so _functools.so _multibytecodec.so  _codecs_jp.so future_builtins.so _multiprocessing.so  _codecs_kr.so grp.so nis.so  _codecs_tw.so _heapq.so operator.so  _collections.so _hotshot.so ossaudiodev.so cPickle.so imageop.so parser.so crypt.so _io.so pyexpat.so cStringIO.so itertools.so Python-2.7.1-py2.7.egg-info  _csv.so _json.so _random.so Host:#

resource.so select.so _sha256.so _sha512.so _sha.so _socket.so spwd.so strop.so _struct.so syslog.so termios.so _testcapi.so time.so unicodedata.so

Excellent, all the shared libraries are built. Let's check the python executable. Host:# cd cawanpython/usr/local/bin cawanpython/usr/local/bin/ / Host:# ls 2to3 idle pydoc python python2.7 python2.7-config python-config smtpd.py Host:# file python python: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1, for GNU/Linux 2.6.12, dynamically linked (uses shared libs), not stripped Host:# Well, it should work by ignoring the error message. Let's verify in MIPS environment. tango3[Python-2.7.1]# cd ./cawanpython/usr/local/ bin/ include/ lib/ share/ tango3[Python-2.7.1]# cd ./cawanpython/usr/local/bin/ tango3[bin]# ls 2to3* pydoc* python-config@ python2.7-config* idle* python* python2.7* smtpd.py* tango3[bin]# ./python Python 2.7.1 (r271:86832, Nov 29 2012, 17:22:08) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> pack Traceback (most recent call last): File "", line 1, in >>> from struct import * >>> pack

>>> Nice, it seems everything is fine now and modules can be imported and run properly. Now, let us take some challenges to cross compile the latest version of python, which is python 3.3.0. We will start by applying the same analogy. Host:# wget http://www.python.org/ftp/py http://www.python.org/ftp/python/3.3.0/Python-3.3.0.tgz thon/3.3.0/Python-3.3.0.tgz For the patch of cross compilation, we need py3k-20121004-CROSS.tgz. Host:# wget http://bugs.python.org/file2 http://bugs.python.org/file27444/py3k-20121004-CROSS.tgz 7444/py3k-20121004-CROSS.tgz Let us start to apply the patch. Host:# tar xzvf py3k-20121004-CROSS.tgz 0001-CROSS-fix-typo-in-thread-AC_CACHE_VAL.patch 0002-CROSS-restore-graminit.-to-source-directory.patch 0003-CROSS-restore-importlib-header-to-source-directory-a.patch 0004-CROSS-restore-AST-to-source-directory.patch 0005-CROSS-revert-issue13150-i.e.-python-solution-with-_s.patch 0006-CROSS-initialise-include-and-library-paths.patch 0007-CROSS-set-_PYTHON_PROJECT_BASE-to-current-build-dir.patch 0008-CROSS-use-_PYTHON_PROJECT_BASE-in-distutils-sysconfi.patch 0009-CROSS-pass-all-top-configure-arguments-to-libffi-con.patch 0010-CROSS-warn-only-if-readelf-is-not-in-host-triplet-fo.patch 0011-CROSS-append-gcc-library-search-paths-instead-to-pre.patch 0012-CROSS-avoid-ncursesw-include-path-hack.patch 0013-CROSS-properly-detect-WINDOW-_flags-for-different-nc.patch 0014-CROSS-use-build-directory-as-root-for-regression-tes.patch 0015-CROSS-test-tools-has-to-depend-only-from-location-of.patch 0016-CROSS-reload-may-fail-with-operation-on-closed-file-.patch Host:# There are 16 of them, let us try it one by one. Host:# cd Python-3.3.0 Host:# patch -p1 < ../0001-CROSS-fix-typo-in-thread-AC_CACHE_VAL.patch ../0001-CROSS-fix-typo-in-thread-AC_CACHE_VAL.patch patching file configure.ac Host:# patch -p1 < ../0002-CROSS-restore-graminit.-to-source-directory.patch ../0002-CROSS-restore-graminit.-to-source-directory.patch patching file Makefile.pre.in Reversed (or (or previously applied) patch patch detected! Assume -R? [n] Apply anyway? [n] Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file Makefile.pre.in.rej Host:# patch -p1 < ../0003-CROSS-restore-imp ../0003-CROSS-restore-importlib-header-to-source-dire ortlib-header-to-source-directory-a.patch ctory-a.patch patching file Makefile.pre.in Hunk #1 succeeded at 586 (offset -5 lines). Hunk #2 succeeded at 1344 (offset -7 lines). Host:# patch -p1 < ../0004-CROSS-restore-AST-to-source-directory.patch ../0004-CROSS-restore-AST-to-source-directory.patch patching file Makefile.pre.in Reversed (or (or previously applied) patch patch detected! Assume -R? [n] Apply anyway? [n] Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file Makefile.pre.in.rej Host:# patch -p1 < ../0005-CROSS-revert-issu ../0005-CROSS-revert-issue13150-i.e.-python-solutione13150-i.e.-python-solution-with-_s.patch with-_s.patch patching file Lib/sysconfig.py patching file Makefile.pre.in Hunk #1 succeeded at 470 with fuzz 1. Host:# patch -p1 < ../0006-CROSS-initialise-include-and-library-paths.patch ../0006-CROSS-initialise-include-and-library-paths.patch patching file setup.py Host:# patch -p1 < ../0007-CROSS-set-_PYTHON ../0007-CROSS-set-_PYTHON_PROJECT_BASE-to-current-bui _PROJECT_BASE-to-current-build-dir.patch ld-dir.patch patching file configure.ac Host:# patch -p1 < ../0008-CROSS-use-_PYTHON ../0008-CROSS-use-_PYTHON_PROJECT_BASE-in-distutils-s _PROJECT_BASE-in-distutils-sysconfi.patch ysconfi.patch patching file Lib/distutils/sysconfig.py Host:# patch -p1 < ../0009-CROSS-pass-all-to ../0009-CROSS-pass-all-top-configure-arguments-to-lib p-configure-arguments-to-libffi-con.patch ffi-con.patch patching file setup.py Hunk #1 FAILED at 1790. 1 out of 1 hunk FAILED -- saving rejects to file setup.py.rej Host:# patch -p1 < ../0010-CROSS-warn-only-i ../0010-CROSS-warn-only-if-readelf-is-not-in-host-tri f-readelf-is-not-in-host-triplet-fo.patch plet-fo.patch

patching file configure.ac Host:# patch -p1 < ../0011-CROSS-append-gcc../0011-CROSS-append-gcc-library-search-paths-instead library-search-paths-instead-to-pre.patch -to-pre.patch patching file setup.py Host:# patch -p1 < ../0012-CROSS-avoid-ncursesw-include-path-hack.patch ../0012-CROSS-avoid-ncursesw-include-path-hack.patch patching file configure.ac Host:# patch -p1 < ../0013-CROSS-properly-de ../0013-CROSS-properly-detect-WINDOW-_flags-for-diffe tect-WINDOW-_flags-for-different-nc.patch rent-nc.patch patching file Include/py_curses.h patching file configure.ac patching file pyconfig.h.in Host:# patch -p1 < ../0014-CROSS-use-build-d ../0014-CROSS-use-build-directory-as-root-for-regress irectory-as-root-for-regression-tes.patch ion-tes.patch patching file Lib/test/regrtest.py Host:# patch -p1 < ../0015-CROSS-test-tools../0015-CROSS-test-tools-has-to-depend-only-from-loca has-to-depend-only-from-location-of.patch tion-of.patch patching file Lib/test/test_tools.py Host:# patch -p1 < ../0016-CROSS-reload-may../0016-CROSS-reload-may-fail-with-operation-on-close fail-with-operation-on-closed-file-.patch d-file-.patch patching file Lib/imp.py Host:# Well, patch 0002, 0004, and 0009 are failed. Let's ignore it first and start our cross compilation process now. Host:# Host:# Host:# Host:# Host:#

./configure make python Parser/pgen mv python hostpython mv Parser/pgen Parser/hostpgen make distclean

Host:# ./configure --host=mips-linux-gnu --target=mips-linux-gnu \ CC="mips-linux-gnu-gcc -EL" CXX="mips-linux-gnu-g++ -EL" AR=mips-linux-gnu-ar \ RANLIB=mips-linux-gnu-ranlib However, we get an error. configure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling Let us try to solve it. Host:# echo ac_cv_file__dev_ptmx=no > config.site Host:# echo ac_cv_file__dev_ptc=no >> config.site Host:# cat ./config.site ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no Host:# export CONFIG_SITE=config.site Host:# set | grep CONFIG_SITE CONFIG_SITE=config.site  _=CONFIG_SITE Host:# Let's run the configure tool again. After a while, the process conpleted. Now, let us try to start the make process. Host:# make HOSTPYTHON=./hostpython HOSTPGEN=./Parser/hostpgen \ BLDSHARED="mips-linux-gnu-gcc BLDSHARED="mips-linux-gnu -gcc -EL -shared" CROSS_COMPILE=mips-linux-gnu- \ CROSS_COMPILE_TARGET=yes After a while, we get an error. case $MAKEFLAGS in *s*) quiet=-q; esac; \ CC='mips-linux-gnu-gcc -EL' LDSHARED='mips-linux-gnu-gcc -EL -shared' OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' \ ./python -E ./setup.py $quiet build ./python: 1: Syntax error: "(" unexpected make: *** [sharedmods] Error 2 The make process should use ./hostpython instead of ./python. Must be something wrong in Makefile, let's check. Host:# nano Makefile Go to line 497 and we should get something like this.

# Build the shared modules sharedmods: $(BUILDPYTHON) $(SYSCONFIGDATA) case $$MAKEFLAGS in *s*) quiet=-q; esac; \ $(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' \ $(PYTHON_FOR_BUILD) $(srcdir)/setup.py $$quiet build Well, it seems the HOSTPYTHON has been replaced by PYTHON_FOR_BUILD. Let us verify it by referring to the Makefile of python 2.7.1. Host:# nano ../../Python-2.7.1/Makefile Go to line 425 and we should get something like this. # Build the shared modules sharedmods: $(BUILDPYTHON) @case $$MAKEFLAGS in \ *s*) $(RUNSHARED) $(RUNSHARED) CC='$(CC)' CC='$(CC)' LDSHARED='$(BLDSHARED)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' OPT='$(OPT)' $(HOSTPYTHON) -E $($ *) $(RUNSHARED) $(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' $(HOSTPYTHON) -E $(sr$ esac Good, we should change the HOSTPYTHON option into PYTHON_FOR_BUILD in python 3.3.0. Let's run the make process again. Host:# make PYTHON_FOR_BUILD=./hostpytho PYTHON_FOR_BUILD=./hostpython n HOSTPGEN=./Parser/hostpgen BLDSHARED="mips-linux-gnu-gcc BLDSHARED="mips-linux-gnu -gcc -EL -shared" CROSS_COMPILE=mips-linux-gnuCROSS_COMPILE_TARGET=yes Fine, after a while, the make process completed. Let's go ahead for make install process, and we should remember to change the HOSTPYTHON into PYTHON_FOR_BUILD. Host:# make install PYTHON_FOR_BUILD=./hostpython BLDSHARED="mips-linux-gnu-gcc -EL -shared" \ CROSS_COMPILE=mips-linux-gnuCROSS_COMPILE=mips-linuxgnu- CROSS_COMPILE_TARGET=yes DESTDIR=$PWD/cawanpython Nice, after a while, the make install process completed. Let's check the destination directory. Host:# cd ./cawanpython/usr/local/ Host:# ls bin include lib share Host:# cd bin Host:# ls 2to3 idle3 pydoc3 python3 2to3-3.3 idle3.3 pydoc3.3 python3.3 Host:# cd ../lib/ Host:# cd python3.3/ Host:# cd lib-dynload/ Host:# ls array.cpython-33m_failed.so atexit.cpython-33m_failed.so audioop.cpython-33m_failed.so  _bisect.cpython-33m_failed.so  _bisect.cpython-33m_faile d.so cmath.cpython-33m_failed.so  _codecs_cn.cpython-33m_failed.so  _codecs_cn.cpython-33m_fa iled.so  _codecs_hk.cpython-33m_failed.so  _codecs_hk.cpython-33m_fa iled.so  _codecs_iso2022.cpython-33m_failed.so  _codecs_iso2022.cpython-3 3m_failed.so  _codecs_jp.cpython-33m_failed.so  _codecs_jp.cpython-33m_fa iled.so  _codecs_kr.cpython-33m_failed.so  _codecs_kr.cpython-33m_fa iled.so  _codecs_tw.cpython-33m_failed.so  _codecs_tw.cpython-33m_fa iled.so  _crypt.cpython-33m_failed.so  _crypt.cpython-33m_failed .so  _csv.cpython-33m_failed.so  _csv.cpython-33m_failed.s o  _ctypes.cpython-33m_failed.so  _ctypes.cpython-33m_faile d.so  _ctypes_test.cpython-33m_failed.so  _ctypes_test.cpython-33m_ failed.so  _datetime.cpython-33m_failed.so  _datetime.cpython-33m_fai led.so  _decimal.cpython-33m_failed.so  _decimal.cpython-33m_fail ed.so  _elementtree.cpython-33m_failed.so  _elementtree.cpython-33m_ failed.so fcntl.cpython-33m_failed.so grp.cpython-33m_failed.so  _heapq.cpython-33m_failed.so  _heapq.cpython-33m_failed .so  _json.cpython-33m_failed.so  _json.cpython-33m_failed. so  _lsprof.cpython-33m_failed.so  _lsprof.cpython-33m_faile d.so

python3.3-config python3.3m

python3.3m-config python3-config

pyvenv pyvenv-3.3

math.cpython-33m_failed.so _md5.cpython-33m_failed.so mmap.cpython-33m_failed.so _multibytecodec.cpython-33m_failed.so _multibytecodec.cpython33m_failed.so _multiprocessing.cpython-33m_failed.so nis.cpython-33m_failed.so nis.cpython-33m_failed.s o parser.cpython-33m_failed.so parser.cpython-33m_faile d.so _pickle.cpython-33m_failed.so _pickle.cpython-33m_fail ed.so _posixsubprocess.cpython-33m_failed.so _posixsubprocess.cpython -33m_failed.so pyexpat.cpython-33m_failed.so pyexpat.cpython-33m_fail ed.so _random.cpython-33m_failed.so _random.cpython-33m_fail ed.so resource.cpython-33m_failed.so resource.cpython-33m_fai led.so _sha1.cpython-33m_failed.so _sha1.cpython-33m_failed .so _sha256.cpython-33m_failed.so _sha256.cpython-33m_fail ed.so _sha512.cpython-33m_failed.so _sha512.cpython-33m_fail ed.so spwd.cpython-33m_failed.so spwd.cpython-33m_failed. so _struct.cpython-33m_failed.so _struct.cpython-33m_fail ed.so syslog.cpython-33m_failed.so syslog.cpython-33m_faile d.so termios.cpython-33m_failed.so _testbuffer.cpython-33m_failed.so _testcapi.cpython-33m_failed.so _testcapi.cpython-33m_fa iled.so time.cpython-33m_failed.so time.cpython-33m_failed. so unicodedata.cpython-33m_failed.so unicodedata.cpython-33m_ failed.so

Host:# The filenames of those shared libraries in lib-dynload directory are really strange. Anyway, we ignore this issue first. Let's test the new python 3.3.0 in MIPS environment. tango3[Python-3.3.0]# cd ./cawanpython/ tango3[cawanpython]# cd .. tango3[Python-3.3.0]# cd ./cawanpython/usr/local/bin/ tango3[bin]# ls 2to3@ pydoc3@ python3.3* pyvenv@ 2to3-3.3* pydoc3.3* python3.3-config@ pyvenv-3.3* idle3@ python3@ python3.3m* idle3.3* python3-config@ python3.3m-config* tango3[bin]# ./python3 Python 3.3.0 (default, Nov 29 2012, 20:02:02) [GCC 4.3.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> pack Traceback (most recent call last): File "", line 1, in NameError: name 'pack' is not defined >>> from struct import * Traceback (most recent call last): File "", line 1, in File "/root/cawan/mips-4.3/bin/te "/root/cawan/mips-4.3/bin/testpython/Python-3.3.0/cawanp stpython/Python-3.3.0/cawanpython/usr/local/lib/ ython/usr/local/lib/ python3.3/struct.py", line 12, in from _struct import * ImportError: No module named '_struct' >>> pack Traceback (most recent call last): File "", line 1, in NameError: name 'pack' is not defined >>> Well, it seems the new python cannot import the struct library. Anyway, it is expected. Let's change the strange filename of struct library into the proper one and try again. tango3[bin]# cd ../lib/python3.3/lib-dynload ../lib/python3.3/lib-dynload/ / tango3[lib-dynload]# ls *struct*  _struct.cpython-33m_failed.so  _struct.cpython-33m_faile d.so tango3[lib-dynload]# cp _struct.cpython-33m_failed.so _struct.so tango3[lib-dynload]# ls _struct.so  _struct.so tango3[lib-dynload]# ls -l _struct.so -rw-r--r-1 root root 86258 Nov 29 2012 _struct.so tango3[lib-dynload]# cd ../../../bin/ tango3[bin]# ./python3 Python 3.3.0 (default, Nov 29 2012, 20:02:02) [GCC 4.3.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> pack Traceback (most recent call last): File "", line 1, in NameError: name 'pack' is not defined >>> from struct import * >>> pack >>> Excellent, the python can import the struct library now and run the built-in command. Let's try to import the socket library to double confirm. tango3[Python-3.3.0]# cd ./cawanpython/usr/local/ tango3[local]# cd bin tango3[bin]# ./python3 Python 3.3.0 (default, Nov 28 2012, 19:31:37) [GCC 4.3.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from socket import *

Traceback (most recent call last): File "", line 1, in File "/root/cawan/mips-4.3/bin/Py "/root/cawan/mips-4.3/bin/Python-3.3.0/cawanpython/usr/l thon-3.3.0/cawanpython/usr/local/lib/ ocal/lib/ python3.3/socket.py", line 47, in import _socket ImportError: No module named '_socket' >>> tango3[bin]# ls ../lib/python3.3/lib-dynload ../lib/python3.3/lib-dynload/*socket* /*socket* ls: ../lib/python3.3/lib-dynload/*socket*: ../lib/python3.3/lib-dynload/*socket*: No such file or directory tango3[bin]# Unfortunately, the shared library of socket doesn't exist at all, so, we even can't rename it from the strange filename. Let's check the make messages again in details. ... ... ... Python build finished, but the necessary bits to build these modules were not found:  _bz2 _dbm _gdbm  _lzma _sqlite3 _ssl  _tkinter readline To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Failed to build these modules:  _bisect _codecs_cn  _codecs_iso2022 _codecs_jp  _codecs_tw _crypt  _ctypes _ctypes_test  _curses_panel _datetime  _elementtree _heapq  _lsprof _md5  _multiprocessing _pickle  _random _sha1  _sha512 _socket  _testbuffer _testcapi atexit audioop cmath fcntl math mmap ossaudiodev parser resource select syslog termios unicodedata zlib ... ... ...

_codecs_hk _codecs_kr _csv _curses _decimal _json _multibytecodec _posixsubprocess _sha256 _struct array binascii grp nis pyexpat spwd time

Well, that's the reason all the filenames of shared libraries in lib-dynload directory are in strange format, they are even not being compiled properly... So, the cross compilation patch for python 3.3.0 might be useless at all. Let us start everything from the begining again, without applying the patch. Host:# tar xzvf Python-3.3.0.tgz Host:# cd Python-3.3.0 Host:# ./configure Host:# make python Parser/pgen Host:# mv python hostpython Host:# mv Parser/pgen Parser/hostpgen Host:# make distclean Host:# echo ac_cv_file__dev_ptmx=yes > ./config.site Host:# echo ac_cv_file__dev_ptc=yes >> ./config.site Host:# export CONFIG_SITE=config.site Host:# set | grep CONFIG_SITE CONFIG_SITE=config.site Host:# ./configure --host=mips-linux-gnu --target=mips-linux-gnu \ CC="mips-linux-gnu-gcc -EL" CXX="mips-linux-gnu-g++ -EL" AR=mips-linux-gnu-ar \ RANLIB=mips-linux-gnu-ranlib RANLIB=mips-linux-gnu-ran lib LD="mips-linux-gnu-ld -EL" Host:# make PYTHON_FOR_BUILD=./hostpython HOSTPGEN=./Parser/hostpgen \ BLDSHARED="mips-linux-gnu-gcc BLDSHARED="mips-linux-gnu -gcc -EL -shared" CROSS_COMPILE=mips-linux-gnu- \

CROSS_COMPILE_TARGET=yes Host:# make install PYTHON_FOR_BUILD=./hostpython \ BLDSHARED="mips-linux-gnu-gcc BLDSHARED="mips-linux-gnu -gcc -EL -shared" CROSS_COMPILE=mips-linux-gnu- \ CROSS_COMPILE_TARGET=yes DESTDIR=$PWD/cawanpython Good, the cross compilation process is completed without applying any patch. However, we still get this message output. Python build finished, but the necessary bits to build these modules were not found:  _bz2 _dbm _gdbm  _lzma _sqlite3 _ssl  _tkinter readline To find the necessary bits, look in setup.py in detect_modules() for the module's name. Failed to build these modules:  _bisect _codecs_cn  _codecs_iso2022 _codecs_jp  _codecs_tw _crypt  _ctypes _ctypes_test  _curses_panel _datetime  _elementtree _heapq  _lsprof _md5  _multiprocessing _pickle  _random _sha1  _sha512 _socket  _testbuffer _testcapi atexit audioop cmath fcntl math mmap ossaudiodev parser resource select syslog termios unicodedata zlib

_codecs_hk _codecs_kr _csv _curses _decimal _json _multibytecodec _posixsubprocess _sha256 _struct array binascii grp nis pyexpat spwd time

Anyway, ignore this and check the lib-dynload directory first. Host:# ls -l total 6200 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 ... ... ... -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 Host:#

./cawanpython/usr/local/lib/python3.3/lib-dynload/ ./cawanpython/usr/local/lib/python3.3/lib-dynload/ root root root root root root root root root root root root

root root root root root root root root root root root root

107510 23385 64169 23615 70333 152927 160944 52443 253712 143311 111884 14570

2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01

17:49 17:50 17:50 17:50 17:50 17:50 17:50 17:50 17:50 17:50 17:50 17:50

root root root root root root root root

root 21492 2012-12-01 2012-12-01 17:50 root 86387 2012-12-01 2012-12-01 17:49 root 23632 2012-12-01 2012-12-01 17:50 root 32439 2012-12-01 2012-12-01 17:50 root 97067 2012-12-01 2012-12-01 17:50 root 133042 2012-12-01 17:50 root 59832 2012-12-01 2012-12-01 17:50 root 800048 2012-12-01 17:50

array.cpython-33m_failed.so atexit.cpython-33m_failed.so atexit.cpython-33m_failed .so audioop.cpython-33m_failed.so audioop.cpython-33m_faile d.so _bisect.cpython-33m_failed.so _bisect.cpython-33m_faile d.so cmath.cpython-33m_failed.so cmath.cpython-33m_failed. so _codecs_cn.cpython-33m_failed.so _codecs_hk.cpython-33m_failed.so _codecs_iso2022.cpython-33m_failed.so _codecs_iso2022.cpython-3 3m_failed.so _codecs_jp.cpython-33m_failed.so _codecs_kr.cpython-33m_failed.so _codecs_tw.cpython-33m_failed.so _crypt.cpython-33m_failed.so _crypt.cpython-33m_failed .so

spwd.cpython-33m_failed.so spwd.cpython-33m_failed.so _struct.cpython-33m_failed.so _struct.cpython-33m_faile d.so syslog.cpython-33m_failed.so syslog.cpython-33m_failed .so termios.cpython-33m_failed.so termios.cpython-33m_faile d.so _testbuffer.cpython-33m_failed.so _testbuffer.cpython-33m_f ailed.so _testcapi.cpython-33m_failed.so time.cpython-33m_failed.so time.cpython-33m_failed.s o unicodedata.cpython-33m_failed.so

So, it seems the Makefile being generated is so far so good, the problem should be in setup.py. After having a brief look to the setup.py, it seems the script can't work in proper once the ssl lib doesn't exist. By referring to the cross compilation patch of python 2.7.1, I created a patch for python 3.3.0, which I named it as python-3.3.0-cross-compile-cawan.patch. python-3.3.0-cross-compil e-cawan.patch. Let's verify.

Host:# make distclean Host:# patch -p1 < python-3.3.0-cross-compil python-3.3.0-cross-compile-cawan.patch e-cawan.patch patching file setup.py Host:# ./configure --host=mips-linux-gnu --target=mips-linux-gnu \ CC="mips-linux-gnu-gcc -EL" CXX="mips-linux-gnu-g++ -EL" AR=mips-linux-gnu-ar \ RANLIB=mips-linux-gnu-ranlib RANLIB=mips-linux-gnu-ran lib LD="mips-linux-gnu-ld -EL" Host:# make PYTHON_FOR_BUILD=./hostpython HOSTPGEN=./Parser/hostpgen \ BLDSHARED="mips-linux-gnu-gcc BLDSHARED="mips-linux-gnu -gcc -EL -shared" CROSS_COMPILE=mips-linux-gnu- \ CROSS_COMPILE_TARGET=yes Host:# make install PYTHON_FOR_BUILD=./hostpython \ BLDSHARED="mips-linux-gnu-gcc BLDSHARED="mips-linux-gnu -gcc -EL -shared" CROSS_COMPILE=mips-linux-gnu- \ CROSS_COMPILE_TARGET=yes DESTDIR=$PWD/cawanpython Excellent, the make message output become this. Python build finished, but the necessary bits to build these modules were not found:  _bz2 _dbm _gdbm  _lzma _sqlite3 _ssl  _tkinter readline To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Failed to build these modules:  _curses _curses_panel  _sha binascii

_md5 zlib

Let's check the generated lib-dynload directory. Host:# ls -l total 6136 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1 -rwxr-xr-x 1

./cawanpython/usr/local/lib/python3.3/lib-dynload/ ./cawanpython/usr/local/lib/python3.3/lib-dynload/ root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root

root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root root

107510 23385 64169 23615 70333 152927 160944 52443 253712 143311 111884 14570 67950 448007 47225 178708 821247 126793 35043 22166 36440 92873 47840 86758 55133 73487 36548 35569 63724 156605 233357 39650 552668 28805 23111 51393 40770 71306

2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01 2012-12-01

18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:23 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22 18:22

array.cpython-33m.so atexit.cpython-33m.so audioop.cpython-33m.so _bisect.cpython-33m.so cmath.cpython-33m.so _codecs_cn.cpython-33m.so _codecs_hk.cpython-33m.so _codecs_iso2022.cpython-33m.so _codecs_iso2022.cpython-3 3m.so _codecs_jp.cpython-33m.so _codecs_kr.cpython-33m.so _codecs_tw.cpython-33m.so _crypt.cpython-33m.so _csv.cpython-33m.so _ctypes.cpython-33m.so _ctypes_test.cpython-33m.so _ctypes_test.cpython-33m. so _datetime.cpython-33m.so _decimal.cpython-33m.so _elementtree.cpython-33m.so fcntl.cpython-33m.so grp.cpython-33m.so _heapq.cpython-33m.so _json.cpython-33m.so _lsprof.cpython-33m.so math.cpython-33m.so mmap.cpython-33m.so _multibytecodec.cpython-33m.so _multibytecodec.cpython-3 3m.so _multiprocessing.cpython-33m.so _multiprocessing.cpython33m.so nis.cpython-33m.so ossaudiodev.cpython-33m.so ossaudiodev.cpython-33m.s o parser.cpython-33m.so _pickle.cpython-33m.so _posixsubprocess.cpython-33m.so _posixsubprocess.cpython33m.so pyexpat.cpython-33m.so _random.cpython-33m.so resource.cpython-33m.so select.cpython-33m.so _sha256.cpython-33m.so _sha512.cpython-33m.so

-rwxr-xr-x -rwxr-xr-x -rwxr-xr-x -rwxr-xr-x -rwxr-xr-x -rwxr-xr-x -rwxr-xr-x -rwxr-xr-x -rwxr-xr-x Host:#

1 1 1 1 1 1 1 1 1

root root root root root root root root root

root 174055 2012-12-01 18:22 _socket.cpython-33m.so root 21492 2012-12-01 2012-12-01 18:22 spwd.cpython-33m.so root 86387 2012-12-01 2012-12-01 18:22 _struct.cpython-33m.so root 23632 2012-12-01 2012-12-01 18:22 syslog.cpython-33m.so root 32439 2012-12-01 2012-12-01 18:22 termios.cpython-33m.so root 97067 2012-12-01 2012-12-01 18:22 _testbuffer.cpython-33m.s _testbuffer.cpython-33m.so o root 133042 2012-12-01 18:22 _testcapi.cpython-33m.so root 59832 2012-12-01 2012-12-01 18:22 time.cpython-33m.so root 800048 2012-12-01 18:22 unicodedata.cpython-33m.so

Nice, the filenames of all shared libraries being generated are in proper format right now. Let's verify in MIPS environment. tango3[Python-3.3.0]# cd ./cawanpython/usr/local/bin tango3[bin]# ./python3 Python 3.3.0 (default, Dec 1 2012, 18:22:07) [GCC 4.3.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> pack Traceback (most recent call last): File "", line 1, in NameError: name 'pack' is not defined >>> from struct import * >>> pack >>> from socket import * >>> As expected, the new python 3.3.0 can import both of the struct and socket modules successfully now. Let's check the network features under python environment. Host:# ifconfig eth9 | grep "inet addr" inet addr:192.168.1.197 Bcast:192.168.1.255 Host:# nc -l -vvv -p 8888 listening on [any] 8888 ...

Mask:255.255.255.0

Well, the host machine is listening to port 8888 now. Let's connect to it from the python environment. tango3[bin]# ./python3 Python 3.3.0 (default, Dec 1 2012, 18:22:07) [GCC 4.3.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import socket >>> cawansock = socket.socket(socket.AF_INET socket.socket(socket.AF_INET, , socket.SOCK_STREAM) >>> cawansock.connect(('192.168.1.197', 8888)) >>> cawansock.send(b'hello cawan') 11 >>> Let's check the host machine now. connect to [192.168.1.197] from (UNKNOWN) [192.168.1.198] 57560 hello cawan Good, the string has been sent from MIPS to host. Let's send something from host to MIPS now. First, instruct MIPS to start receive. >>> cawandata=cawansock.recv(3 cawandata=cawansock.recv(32) 2) Then, from host machine, type something and press enter. cawan reply Let's check for anything received in MIPS. >>> print(cawandata) b'cawan reply\n'

Nice, the string received. Now, disconnect the socket. >>> cawansock.close() >>> In host machine, it backs to the prompt. cawan reply sent 24, rcvd 11 Host:# So, the python 3.3.0 is running well in our MIPS platform now. ///////////////////python-3.3.0-cross-compile-cawan.patch/////////////////// ///////////////////python-3.3.0-cross-compile-cawan.pa tch/////////////////// begin-base64 644 . H4sIAMLsuVAAA+1Y62+bSBDv18tfMZeoAmrDgV9xfJdT0rwaXfNQnFNVnSq0 xmubBu8idknq//5mFzsG7DzatD2dxHwIBJbfvH4zO+t4Jiec2U2n6bh2kHAh 7IBP4zCidkDuCHNiIoPJqxeJi9JptfQVpXT1vNZ265XXcNudZrvVauNzr9Nq tl6B+zK1z5NUSJIAvEo4l4+te+r9/1Rs2wbym6AyjZ149kvD9Rq2u2M3dsDr 9Fy313YddyFQc7uuu1Gr1WBQ+kR95YHX7XmNXnN75ZO9PbCbzUa9AzV16cLe 3gbkRdBo5BDGeMoCahof9q/OT89PeiBuwjgO2RjCacwTCcGEBjcw4gkczMZ3 IbMHRNAhbL4Wm0YRUclroF+kw8iUWsWXCRqfsI3a4v9wBFw4lN2GCWfOmErT OLi66Pf9g4uzy9P3R/71/tXJ0bVhwe4uGDMqjN7y4xzg/f+o1x9hESndsKvA sY4mzmceMnON54M0jIZ+FA7qa16iOX4ezyw+TaNIP7131bKyeLe6GOhas+2W 4i2TWa+oBoPrRJwM/eEMEcLgHqtecMTK+xfQWMKpzspRkmBCiIC7CSLbKw6M CAIMHRLHlA2Xdq5ZuWTAmzdv4J4FiVKvWKDyDCLEFXNCaGpIo5iM52fz113Y xGxulrL5cnvKaDkxIItHD14LAymaCzbGz6pDRG9ptNssUZYIQbEAGJeZaSGL IxLQ4iJVDRmURB054ok4CiUqMov51Dzp7Oi63PbqnrdSmAtBoIRO+S0117Ah x4h9KZNwkEqqSdFbBSvFNWVkEFGQHDJ4UPACzHDMeEKHlmEVU0MjQZ/O1n2m dIKyeC8zVAc0EO4oRpqgxlCCsYpYStqABiQVVH1FEgr5nRIRMY9PISxpXyuX UW8lkr6cxRkfMJMDzKOYIaHxechG3LQe7Z0Fpi5ZifopEyFnWUwMnfuu59a3 odb1tvGSTz2y2+dYrkJE/i1N0AL3i+vuuNvY05erJhg+n7DZYqkyFD/AehAQ Cs3Vc84wYmyoX2CDK7woIaVCcSEHZq4oQKRi21gjebv/3C27Yj2VqCe+184c E6QhpmHpAPacsrHlJjta52Uvy0Njp+61MBHNTr3ZKGRiCz5gAKM7MhMwn8xA TihSEdsEQ4rgnwvE6vffq9CSWyS7LikzFCKlsOW1OjtNy8kjnkpDoDHJFEtN aIcQEfjgMw1wi+VDqpCQNTMwW67dduGvtxCj61M+TCNaz0NxFs1A7R1YYNoW EsiURPgQq2WIau+zhQQUiz0AjhZsNA1fTEij3THq8I+R3WZqnMD4VH8y2UOq AMXuP8aEiAkSzJkYnyywnqe37TUWevH2J+mdDttaKV5/nqfewk/vu+isLato 69sSu/z+q3U/S/PDqf2hmh9I7o/2dn16X6C1MBvP56ZD/8Pp9Tv/8uPh0du/ TwBnPtXI1za1Qo/dgmtsL8rMeQdR21JEp5RJoTtP/92+B8oCbHRjnoRyMnVK 2/4jvi9cX3peiFvOAkzKAxZc9ffhkEgCfRqkaADuu6cscODssF1GOsOeScbU PgzHVODAs7C4jtEUAQ4/atRgcHV8AF6z4TmgdJdBGA0UTDKbzztomRPoTqzu JnrGwG00SlVfndCEPjcemn+Pb3GCpwlqx/21SNM6GNqMAnMeYc0CYA1n1J6p Sn7tDNFVM8Sz6GVeJ7iD4U0e5o+16NYK5xab4pBTwQyJ1/m+mTKJs7Hr7Dhd jAVOdEYUwSBRQxJGBvgd01x8PgEfanIrNPyWfpVFFvJ77pl+qYhLJMQJvw1x x8b9WYQCIdHTMJCIi+yyo/CG4oA4JehzIJCLH3kKd2EU5fEQYoDlO4M7XKZG cZIkhI2pPmejChyTkehqkOMjFcOpWjOguVmDMz3GeNvdjjpzel3Pq+88eJZY K6NR6AecjcKxqghrzclhlPAp+iZkivkTzjBMfHW3+GlgeqNOOqufZc9Nha+P 2fidtbrxZap9kow1r/GaeY9XLGYcv7MF+tg9X3tLEnPz4OL8+PTExwNlf9PK jlnmGviSIOlN07DtCRdy11AaUJGlKY9PtZnLx9an1XLc+r72ruJ/b4OL9n6l gZuwWRi2l1Ze4rEY6DSWMzg4fr9/0oflQU2V9ecUmZtVnO7zCRVppM5D68Dm CEjuS/2rqCF+B3usnLQvGnokntOeY8ENnY3/+re7SiqppJJKKqmkkkoqqaSS SiqppJJKKnlK/gUxgM3OACgAAA== ==== ///////////////////python-3.3.0-cross-compile-cawan.pa ///////////////////python -3.3.0-cross-compile-cawan.patch/////////////////// tch/////////////////// To retrieve the patch file, simply copy the content from "begin-base64 644 ." to "====" and paste it in a file then name it as cawan. After that, under the same directory, run the following commands. Host:# uudecode cawan -o cawan.tgz Host:# tar xzvf cawan.tgz ////////////////////////////////////////////////////// ///////////////////////// /////////////////////////////////////////////////// //////////////////////