2. Core TensorFlow

CoreTensorFlow

LakLakshmanan

Learnhowto... Writelazyevaluationand imperativeprograms

Learnhowto... Writelazyevaluationand imperativeprograms Workwithgraphs,sessions, andvariables

Learnhowto... Writelazyevaluationand imperativeprograms Workwithgraphs,sessions, andvariables VisualizeTensorFlowgraphs

Learnhowto... Writelazyevaluationand imperativeprograms Workwithgraphs,sessions, andvariables VisualizeTensorFlowgraphs DebugTensorFlowprograms

WhatisTensorFlow

LakLakshmanan

TensorFlowisan open-source high-performancelibrary high-performance library fornumerical for numerical computationthatuses computation thatuses directedgraphs

Nodesrepresent mathematicaloperations



Edgesrepresent arraysofdata




Edgesrepresent arraysofdata

AtensorisanN-dimensionalarrayofdata

Rank0 Tensor scalar



Rank1 Tensor vector




Rank2 Tensor matrix




Rank2 Tensor matrix

Rank3 Tensor

Rank4 Tensor

BenefitsofaDirectedGraph

LakLakshmanan

TensorFlowgraphsareportablebetween differentdevices

CPUs

GPUs

TensorFlowLiteprovideson-deviceinferenceofML modelsonmobiledevicesandisavailablefora varietyofhardware

Trainoncloud

TensorFlowLiteprovideson-deviceinferenceofML modelsonmobiledevicesandisavailablefora varietyofhardware

Trainoncloud

RuninferenceoniOS, Android,RaspberryPi,etc.

TensorFlowevensupports federatedlearning

https://research.googleblog.com/2017/04/federated-lear ning-collaborative.html







TensorFlowispopularamongbothdeeplearning researchersandmachinelearningengineers

TensorFlowispopularamongbothdeeplearning researchersandmachinelearningengineers

#1repositoryfor“machinelearning”categoryonGitHub

TensorFlowispopularamongbothdeeplearning researchersandmachinelearningengineers c e r u s o e n p o

#1repositoryfor“machinelearning”categoryonGitHub

TensorFlowAPIhierarchy

LakLakshmanan

TensorFlowtoolkithierarchy RunTFat scalewith CMLE

tf.estimator

High-levelAPIfordistributed training

tf.layers,tf.losses,tf.metrics

Componentsusefulwhen buildingcustomNNmodels

CoreTensorFlow(Python)

PythonAPIgivesyoufull control

CoreTensorFlow(C++)

C++APIisquitelowlevel

CPU

GPU

TPU

Android

TFrunsondifferenthardware

e n i g n E L M d u o l C

TensorFlowtoolkithierarchy

tf.estimator








CPU

GPU

TPU

Android




tf.estimator








CPU

GPU

TPU

Android




tf.estimator








CPU

GPU

TPU

Android




tf.estimator








CPU

GPU

TPU

Android




tf.estimator








CPU

GPU

TPU

Android



TensorFlowtoolkithierarchy RunTFat scalewith CMLE

tf.estimator








CPU

GPU

TPU

Android



ThePythonAPIletsyou buildandrunDirected Graphs

... c = tf.add(a, b)

Build

ThePythonAPIletsyoubuildandrunDirectedGraphs a


Build

b

add c

Tensors

Operations



Build

b

add

Run session = tf.Session() numpy_c = session.run(c, feed_dict= ….)

c

Tensors

Operations



Build

b

add


c

Tensors

Operations



Build

b

add


c

Tensors

Operations



Build

b

add


c

Tensors

Operations

TensorFlowdoeslazyevaluation: youneedtorunthegraphtoget results* Build

a = tf.constant([5, 3, 8]) b = tf.constant([3, -1, 2]) c = tf.add(a, b) print c

Tensor("Add_7:0", shape=(3,), dtype=int32) with tf.Session() as sess: result = sess.run(c) print result [8 2 10]

Run

*tf.eager, however,allows youtoexecute operations imperativelylink

TensorFlowdoeslazyevaluation:youneedtorunthe graphtogetresults* numpy a = np.array([5, 3, 8]) b = np.array([3, -1, 2]) c = np.add(a, b) print c [8 2 10]

*tf.eager,however,allows youtoexecuteoperations imperativelylink

TensorFlow Build



Run



TensorFlow Build



Run



TensorFlow Build



Run



TensorFlow Build



Run

GraphandSession

LakLakshmanan

Graphscanbeprocessed,compiled,remotely executed,andassignedtodevices Data:tensors(N-dimensionalarrays) Nodes:operations (ops)ontensors

biases

Add

weights MatMul examples

Relu

Graphscanbeprocessed,compiled,remotely executed,andassignedtodevices

biases Add


send

recv

Graphscanbeprocessed,compiled,remotely executed,andassignedtodevices

Inadditiontocomputation(e.g.,add, matmul),constants,variables,logging,and controlflowarealsoOps

biases

Add


send

recv

SessionallowsTensorFlowtocacheanddistribute computation Python front end

...

Session

CoreTensorFlowExecutionSystem Ops

add

mul

Print

reshape

...

Kernels

CPU

GPU

Android

iOS

...

ExecuteTensorFlow graphsbycallingrun()on atf.Session import tensorflow as tf x = tf.constant([3, 5, 7]) y = tf.constant([1, 2, 3]) z = tf.add(x, y) with tf.Session() as sess: print sess.run(z) [4 7 10]

ExecuteTensorFlowgraphsbycallingrun() onatf.Session import tensorflow as tf x = tf.constant([3, 5, 7]) y = tf.constant([1, 2, 3]) z = tf.add(x, y) with tf.Session() as sess: print sess.run(z) [4 7 10] [4 7 10]

x

y


x

y

tf.add

z


x

y

tf.add

z

Evaluatingatensorisa shortcuttocallingrun()on thegraph’sdefaultsession import tensorflow as tf x = tf.constant([3, 5, 7]) y = tf.constant([1, 2, 3]) z = tf.add(x, y) with tf.Session() as sess: print z.eval() [4 7 10] [4 10]

Itispossibletoevaluatea listoftensors Shortcuts tocommon arithmetic operators

run() acceptsa listof tensors

Itispossibletoevaluatea listoftensors Shortcuts tocommon arithmetic operators

run() acceptsa listof tensors

TensorFlowinEagermodemakesiteasiertotryout things,butisnotrecommendedforproductioncode Importtf.eager Call exactly once

Notethevalue beingprinted






Youcanwritethegraphoutusing tf.summary.FileWriter Namethe tensors andthe operations Writethe session graphtoa summary directory

Youcanwritethegraphoutusing tf.summary.FileWriter Namethe tensors andthe operations Writethe session graphtoa summary directory

Thegraphcanbe visualizedinTensorBoard

Thegraphcanbevisualizedin TensorBoard

Youcanalsowritetogs:// andstartTensorBoard fromCloudShell

1

Runthefollowingcommandin CloudShelltostart TensorBoard: tensorboard--port8080--logdir gs://${BUCKET}/${SUMMARY_DIR}


2

Toopenanewbrowser window,selectPreviewon port8080fromtheWeb previewmenuinthetop-right corneroftheCloudShell toolbar. Inthenewwindow,youcanuse TensorBoardtoseethetraining summaryandthevisualized networkgraph.


3

PressControl+Ctostop TensorBoardinCloudShell. https://cloud.google.com/ml-eng ine/docs/distributed-tensorflowmnist-cloud-datalab

TensorandVariable

LakLakshmanan

AtensorisanN-dimensionalarrayofdata Common name

Rank (Dimension)

Example

Shape of example

Scalar

0

x = tf.constant(3)

()

Vector

1

x = tf.constant([3, 5, 7])

(3,)

Matrix

2

x = tf.constant([[3, 5, 7], [4, 6, 8]])

(2,3)

3D Tensor

3

tf.constant([[[3, 5, 7],[4, 6, 8]], [[1, 2, 3],[4, 5, 6]] ])

(2,2,3)

nD Tensor

n

x1 x2 x3 x4

(3,) (2,3) (4,2,3) (2,4,2,3)

= = = =

tf.constant([2, 3, 4]) tf.stack([x1, x1]) tf.stack([x2, x2, x2, x2]) tf.stack([x3, x3])


Rank (Dimension)

Example

Shape of example

Scalar

0

x = tf.constant(3)

()

Vector

1


(3,)

Matrix

2

x = tf.constant([[3, 5, 7], [4, 6, 8]])

(2,3)

3D Tensor

3

tf.constant([[[3, 5, 7],[4, 6, 8]], [[1, 2, 3],[4, 5, 6]] ])

(2,2,3)

nD Tensor

n

x1 x2 x3 x4

(3,) (2,3) (4,2,3) (2,4,2,3)

= = = =



Rank (Dimension)

Example

Shape of example

Scalar

0

x = tf.constant(3)

()

Vector

1


(3,)

Matrix

2

x = tf.constant([[3, 5, 7], [4, 6, 8]])

(2,3)

3D Tensor

3

tf.constant([[[3, 5, 7],[4, 6, 8]], [[1, 2, 3],[4, 5, 6]] ])

(2,2,3)

nD Tensor

n

x1 x2 x3 x4

(3,) (2,3) (4,2,3) (2,4,2,3)

= = = =



Rank (Dimension)

Example

Shape of example

Scalar

0

x = tf.constant(3)

()

Vector

1


(3,)

Matrix

2

x = tf.constant([[3, 5, 7], [4, 6, 8]])

(2,3)

3D Tensor

3

tf.constant([[[3, 5, 7],[4, 6, 8]], [[1, 2, 3],[4, 5, 6]] ])

(2,2,3)

nD Tensor

n

x1 x2 x3 x4

(3,) (2,3) (4,2,3) (2,4,2,3)

= = = =



Rank (Dimension)

Example

Shape of example

Scalar

0

x = tf.constant(3)

()

Vector

1


(3,)

Matrix

2

x = tf.constant([[3, 5, 7], [4, 6, 8]])

(2,3)

3D Tensor

3

tf.constant([[[3, 5, 7],[4, 6, 8]], [[1, 2, 3],[4, 5, 6]] ])

(2,2,3)

nD Tensor

n

x1 x2 x3 x4

(3,) (2,3) (4,2,3) (2,4,2,3)

= = = =



Rank (Dimension)

Example

Shape of example

Scalar

0

x = tf.constant(3)

()

Vector

1


(3,)

Matrix

2

x = tf.constant([[3, 5, 7], [4, 6, 8]])

(2,3)

3D Tensor

3

tf.constant([[[3, 5, 7],[4, 6, 8]], [[1, 2, 3],[4, 5, 6]] ])

(2,2,3)

nD Tensor

n

x1 x2 x3 x4

(3,) (2,3) (4,2,3) (2,4,2,3)

= = = =


Tensorscanbesliced




Quiz: (1) What would x[1, :] do?

(2) How about

x[1, 0:2]?


Quiz: (1) What would x[1, :] do? [4,6,8] (2) How about [4,6,8]

x[1, 0:2]?

Tensorscanbereshaped







Avariableisatensorwhosevalueisinitializedandthen typicallychangedastheprogramruns

Createvariable, specifyinghowto initandwhether itcanbetuned “Trainingloop”of 5updatesto weights















Avariableisatensorwhosevalueisinitializedandthen typicallychangedastheprogramruns Multiplying[1,2] x[2,3]yieldsa [1,3]matrix Initializeall variables Print[1,3] matrixateach ofthe5 iterations




Createvariable, specifyinghowtoinitand whetheritcanbetuned

“Trainingloop”of5 updatestoweights

Multiplying[1,2]x[2,3] yieldsa[1,3]matrix Initializeallvariables





















Placeholdersallowyoutofeedinvalues,suchasby readingfromatextfile


Tensor("Placeholder :0", dtype=float32)





[

4.

8.

12.]

LAB Writinglow-level TensorFlowprograms

LakLakshmanan

Lab:Writinglow-levelTensorFlowprograms Inthislab,youwilllearnhowtheTensorFlowPythonAPIworks 1.

Building a graph

2.

Running a graph

3.

Feeding values into a graph

4.

Findingtheareaofatrianglebyusing TensorFlow

DebuggingTensorFlow programs

LakLakshmanan

DebuggingTensorFlowprogramsissimilarto debugginganypieceofsoftware 1

2

ReadError Messages

Isolatethe methodin question

3

Send made-up dataintothe method

4

Knowhow tosolve common problems

Readerrormessagesto understandtheproblem 1

in some_method(data) 2 a = data[:,0:2] 3 c = data[:,1] ----> 4 s = (a + c) 5 return tf.sqrt(tf.matmul( tf.sqrt(tf.matmul(s, s, tf.transpose(s)))

Readerrormessagesto understandtheproblem 1

output: ValueError: Dimensions must be equal, but are 2 and 4 for 'add' (op: 'Add') with input shapes: [4,2], [4].

Isolatethemethodin question 2

def some_method(data): a = data[:,0:2] c = data[:,1] s = (a + c) return tf.sqrt(tf.matmul(s, tf.transpose(s)))

Calltheproblematic methodwithfakedata 3

def some_method(data): a = data[:,0:2] print a.get_shape() c = data[:,1] print c.get_shape() s = (a + c) return tf.sqrt(tf.matmul(s, tf.transpose(s))) with tf.Session() as sess: fake_data = tf.constant([ [5.0, 3.0, 7.1], [2.3, 4.1, 4.8], [2.8, 4.2, 5.6], [2.9, 8.3, 7.3] ]) print sess.run(some_method(fake_data))


def some_method(data): a = data[:,0:2] print a.get_shape() c = data[:,1] print c.get_shape() s = (a + c) return tf.sqrt(tf.matmul(s, tf.transpose(s))) with tf.Session() as sess: fake_data = tf.constant([ [5.0, 3.0, 7.1], [2.3, 4.1, 4.8], [2.8, 4.2, 5.6], [2.9, 8.3, 7.3] ]) print sess.run(some_method(fake_data))


def some_method(data): a = data[:,0:2] (4, 2) print a.get_shape() Whydoesthe c = data[:,1] additionfail? print c.get_shape() (4,) s = (a + c) return tf.sqrt(tf.matmul(s, tf.transpose(s))) with tf.Session() as sess: fake_data = tf.constant([ [5.0, 3.0, 7.1], [2.3, 4.1, 4.8], [2.8, 4.2, 5.6], [2.9, 8.3, 7.3] ]) print sess.run(some_method(fake_data))

Knowhowtosolve commonproblems 4

Tensorshape Scalar-vectormismatch Datatypemismatch

Themostcommonproblemtendstobetensorshape def some_method(data): a = data[:,0:2] print a.get_shape() (4, 2) c = data[:,1] Whydoestheadditionfail? (4,) print c.get_shape() s = (a + c) return tf.sqrt(tf.matmul(s, tf.transpose(s))) with tf.Session() as sess: fake_data = tf.constant([ [5.0, 3.0, 7.1], [2.3, 4.1, 4.8], [2.8, 4.2, 5.6], [2.9, 8.3, 7.3] ]) print sess.run(some_method(fake_data))

Themostcommonproblemtendstobetensorshape c = data[:,1:3]

output:

(4, 2) [[ 12.88448715 [ 11.87813091 [ 12.44909573 [ 15.72132301

Add an assert

11.87813091 10.96220779 11.48999596 14.50930595

12.44909573 11.48999596 12.04325485 15.20789242

15.72132301] 14.50930595] 15.20789242] 19.20416641]]

Shapeproblemsalsohappenbecauseofbatchsizeor becauseyouhaveascalarwhenavectorisneeded(or viceversa) Tensor("Placeholde r_4:0", shape=(?, 3), dtype=float32)

ValueError: Cannot feed value of shape (3,)

output

Tensor("Placeholder_4:0", shape=(?, 3), dtype=float32)



output




output




output




output


Shapeproblemscanoften befixedusing... 1. tf.reshape() 2. tf.expand_dims() 3. tf.slice() 4. tf.squeeze()

tf.expand_dimsinsertsadimensionof1intoa tensor’sshape tf.expand_dims() output x = tf.constant([[3, 2], [4, 5], [6, 7]]) print "x.shape", x.shape expanded = tf.expand_dims(x, 1) print "expanded.shape", expanded.shape

x.shape (3, 2) expanded.shape (3, 1, 2)

with tf.Session() as sess: print "expanded:\n", expanded.eval()

[[4 5]]

expanded: [[[3 2]]

[[6 7]]]




[[4 5]]

expanded: [[[3 2]]

[[6 7]]]




[[4 5]]

expanded: [[[3 2]]

[[6 7]]]




[[4 5]]

expanded: [[[3 2]]

[[6 7]]]

tf.sliceextractsaslicefromatensor tf.slice()output x = tf.constant([[3, 2],

x.shape (3, 2)

[4, 5],

sliced.shape (2, 1)

[6, 7]])

sliced:

print "x.shape", x.shape sliced = tf.slice(x, [0, 1], [2, 1]) print "sliced.shape", sliced.shape with tf.Session() as sess: print "sliced:\n", sliced.eval()

[[2] [5]]


x.shape (3, 2)

[4, 5],

sliced.shape (2, 1)

[6, 7]])

sliced:


[[2] [5]]


x.shape (3, 2)

[4, 5],

sliced.shape (2, 1)

[6, 7]])

sliced:


[[2] [5]]

tf.squeezeremovesdimensionsofsize1fromthe shapeofatensor t [[[1] [2] [3] [4]]

output

[[5] [6] [7] [8]]] t squeezed [[1 2 3 4] [5 6 7 8]]

tf.squeezeremovesdimensionsofsize1fromthe shapeofatensor t [[[1] [2] [3] [4]]

output

[[5] [6] [7] [8]]] t squeezed [[1 2 3 4] [5 6 7 8]]

Anothercommonproblem isdatatype

ValueError: Tensor conversion requested dtype float32 for Tensor with dtype int32: 'Tensor("Const_34:0", shape=(2, 3), dtype=int32)'

Thereasonisbecausewearemixingtypes def some_method(a, b): s = (a + b) return tf.sqrt(tf.matmul(s, tf.transpose(s)))

with tf.Session() as sess: fake_a = tf.constant([ [5.0, 3.0, 7.1], [2.3, 4.1, 4.8], ]) fake_b = tf.constant([ [2, 4, 5], [2, 8, 7] ]) print sess.run(some_method(fake_a, fake_b))

Addinga tensoroffloats toatensorof intswon’twork

Thereasonisbecausewearemixingtypes def some_method(a, b): s = (a + b) return tf.sqrt(tf.matmul(s, tf.transpose(s)))

with tf.Session() as sess: fake_a = tf.constant([ [5.0, 3.0, 7.1], [2.3, 4.1, 4.8], ]) fake_b = tf.constant([ [2, 4, 5], [2, 8, 7] ]) print sess.run(some_method(fake_a, fake_b))

Addinga tensoroffloats toatensorof intswon’twork

Onesolutionistodoacast withtf.cast() b = tf.cast(b,tf.float32)

Onesolutionistodoacast withtf.cast() output [[ 15.63361835 [ 16.04929924

16.04929924] 17.43960953]]

Todebugfull-blown programs,therearethree methods

Todebugfull-blown programs,therearethree methods tf.Print()


tfdbg


tfdbg

TensorBoard

Changelogginglevelfrom WARN

tf.Print()canbeusedtologspecifictensorvalues

%bash python xyz.py

Output: [[ [ https://www.tensorflow.org/programmers_guide/debugger

nan nan

nan] 1.43365264]]


%bash python xyz.py


nan nan

nan] 1.43365264]]


%bash python xyz.py


nan nan

nan] 1.43365264]]

TensorFlowhasadynamic,interactivedebugger(no easywaytouseitfromDatalabcurrently)

# in a Terminal window python xyz.py --debug

https://www.tensorflow.org/programmers_guide/debugger

UseTensorFlowdebuggertostepthroughcode

UseTensorFlowdebuggertostepthroughcode

2. Core TensorFlow

Recommend Documents