Check and fix tensor and scalar type promotion

[chenwhql]

PR types

Function optimization

PR changes

APIs

Describe

Check and fix the type promotion problem when calculating paddle tensor and scalar.

For the calculation between tensor and scalar, paddle was originally partially supported. In some scenarios, there are problems with the results and types of calculate result.

This PR add some unittests for each binary operation, including:

• + (add): tensor + scalar
• - (sub & rsub): tensor - scalar, scalar - tensor
• * (mul): tensor * scalar
• / (div & rdiv): tensor / scalar, scalar / tensor
• ** (pow & rpow): tensor ** scalar, scalar ** tensor
• // (floordiv): tensor // scalar
• % (mod): tensor % scalar

Fix several error cases

• Case1: tensor(int) +, -, * scalar(float)
  • original result: tensor(int) or error
  • new result: tensor(float)

a = paddle.ones([2, 2, 2], dtype='int64')
b = 1.0
a + b

# Original:
# Tensor(shape=[2, 2, 2], dtype=int64, place=CUDAPlace(0), stop_gradient=True,
#        [[[2, 2],
#          [2, 2]],

#         [[2, 2],
#          [2, 2]]])
# New:
# Tensor(shape=[2, 2, 2], dtype=float32, place=CUDAPlace(0), stop_gradient=True,
#        [[[2., 2.],
#          [2., 2.]],

#         [[2., 2.],
#          [2., 2.]]])

a = paddle.ones([2, 2, 2], dtype='int64')
b = 1.5
a + b

# Original:
# AssertionError: float value 1.5 cannot convert to integer
# New:
# Tensor(shape=[2, 2, 2], dtype=float32, place=CUDAPlace(0), stop_gradient=True,
#        [[[2.50000000, 2.50000000],
#          [2.50000000, 2.50000000]],

#         [[2.50000000, 2.50000000],
#          [2.50000000, 2.50000000]]])

• Case2: tensor(int) / scalat(int or float)
  • original result: tensor(int) or error
  • new result: tensor(float)

a = paddle.ones([2, 2, 2], dtype='int64')
b = 2
a / b

# Original:
# Tensor(shape=[2, 2, 2], dtype=int64, place=CUDAPlace(0), stop_gradient=True,
#        [[[0, 0],
#          [0, 0]],

#         [[0, 0],
#          [0, 0]]])
# New:
# Tensor(shape=[2, 2, 2], dtype=float32, place=CUDAPlace(0), stop_gradient=True,
#        [[[0.50000000, 0.50000000],
#          [0.50000000, 0.50000000]],

#         [[0.50000000, 0.50000000],
#          [0.50000000, 0.50000000]]])

a = paddle.ones([2, 2, 2], dtype='int64')
b = 0.5
a / b

# Original:
# Error: /work/paddle/paddle/fluid/operators/elementwise/elementwise_op_function.cu.h:66 Assertion `b != 0` failed. InvalidArgumentError: Integer division by zero encountered in divide. Please check.
# New:
# Tensor(shape=[2, 2, 2], dtype=float32, place=CUDAPlace(0), stop_gradient=True,
#        [[[2., 2.],
#          [2., 2.]],

#         [[2., 2.],
#          [2., 2.]]])

• Case3: scalar(int or float) / tensor(int)
  • original result: tensor(int)
  • new result: tensor(float)

a = 1
b = paddle.full([2, 2, 2], 2, dtype='int64')
a / b

# Original:
# Tensor(shape=[2, 2, 2], dtype=int64, place=CUDAPlace(0), stop_gradient=True,
#        [[[0, 0],
#          [0, 0]],

#         [[0, 0],
#          [0, 0]]])
# New:
# Tensor(shape=[2, 2, 2], dtype=float32, place=CUDAPlace(0), stop_gradient=True,
#        [[[0.50000000, 0.50000000],
#          [0.50000000, 0.50000000]],

#         [[0.50000000, 0.50000000],
#          [0.50000000, 0.50000000]]])

a = 1.0
b = paddle.full([2, 2, 2], 2, dtype='int64')
a / b

# Tensor(shape=[2, 2, 2], dtype=int64, place=CUDAPlace(0), stop_gradient=True,
#        [[[0, 0],
#          [0, 0]],

#         [[0, 0],
#          [0, 0]]])
# New:
# Tensor(shape=[2, 2, 2], dtype=float32, place=CUDAPlace(0), stop_gradient=True,
#        [[[0.50000000, 0.50000000],
#          [0.50000000, 0.50000000]],

#         [[0.50000000, 0.50000000],
#          [0.50000000, 0.50000000]]])

• Case4: tensor(int) **, % scalar(float)
  • original result: tensor(int)
  • new result: tensor(float)

a = paddle.full([2, 2, 2], 2, dtype='int64')
b = 3.0
a ** b

# Original:
# Tensor(shape=[2, 2, 2], dtype=int64, place=CUDAPlace(0), stop_gradient=True,
#        [[[8, 8],
#          [8, 8]],

#         [[8, 8],
#          [8, 8]]])
# New:
# Tensor(shape=[2, 2, 2], dtype=float32, place=CUDAPlace(0), stop_gradient=True,
#        [[[8., 8.],
#          [8., 8.]],

#         [[8., 8.],
#          [8., 8.]]])

Performance: Almost no effect

Test Case：

• python2: original impl, result's dtype error, no cast
• python3: new impl, correct result, add cast

import time

import paddle

paddle.set_device('cpu') # gpu

a = paddle.ones([512, 3, 124, 124], dtype='int32')  
b = 2.0

s_time = time.time()
for i in range(1000):
    a = a + b
    a.numpy()
e_time = time.time()

avg_time = (e_time - s_time) / 1000

print(avg_time)

CPU Time（with t.numpy()）：

original: 0.145429956198 (python2)
new: 0.13928969502449035 (python3)

λ yq01-gpu-255-137-12-00 /work/scripts/type_promote {master} python2 profile.py 
grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
0.145429956198

λ yq01-gpu-255-137-12-00 /work/scripts/type_promote {master} python profile.py 
grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
0.13928969502449035

GPU Time（with t.numpy()）：

original: 0.115635383129 (python2)
new: 0.11310314869880676 (python3)

λ yq01-gpu-255-137-12-00 /work/scripts/type_promote {master} python2 profile.py 
grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
W1028 06:29:55.154489 26554 device_context.cc:338] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 10.1, Runtime API Version: 10.1
W1028 06:29:55.162390 26554 device_context.cc:346] device: 0, cuDNN Version: 7.6.
0.115635383129

λ yq01-gpu-255-137-12-00 /work/scripts/type_promote {master} python profile.py 
grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
W1028 06:24:19.684199 26421 device_context.cc:338] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 10.1, Runtime API Version: 10.1
W1028 06:24:19.691251 26421 device_context.cc:346] device: 0, cuDNN Version: 7.6.
0.11310314869880676

Function that may need support

• Our // (floordiv) does not support floating type, no related kernel

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[jzhang533]

lgtm

[ForFishes]

LGTM

[XiaoguangHu01]

LGTM

[zhiqiu]

opetator -> operator?

[chenwhql]

thx, I'll fix it in next PR